• Registered: 2012-08-28

Topic: Setting up a mirror on Ubuntu 12.04 - some issues

So I thought I would have a go at setting up a mirror on a slim install of 12.04.  This particular install is used as a downloader, running Sickbeard, headphones etc.  Very typical.  But seeing some perl issues that I'm pretty sure are simple ...  Basically, the "carton install --deployment" pass does not seem to see existing modules already installed.  Carton was installed with

   sudo cpan Carton

and choosing "sudo" as the method

As an example, I've installed Try::Tiny directly, with

   sudo apt-get install libtry-tiny-perl

which installs version 0.11-1.  Yet the "carton install --deployment" insists on installing it's own.  They're exactly the same ...

  :/usr/local/musicbrainz-server$ l local/lib/perl5/Try/Tiny.pm /usr/share/perl5/Try/Tiny.pm
  -r--r--r-- 1 downloads downloads 15252 Aug 30  2011 local/lib/perl5/Try/Tiny.pm
  -rw-r--r-- 1 root      root      15252 Aug 30  2011 /usr/share/perl5/Try/Tiny.pm

  :/usr/local/musicbrainz-server$ md5sum local/lib/perl5/Try/Tiny.pm /usr/share/perl5/Try/Tiny.pm
  9953b73a1b55deebc51ecbb424930671  local/lib/perl5/Try/Tiny.pm
  9953b73a1b55deebc51ecbb424930671  /usr/share/perl5/Try/Tiny.pm

It would be much nicer to use actual ubuntu/debian packages - 90% of them are there, so will be updated and maintained normally rather than pulling them in with cpan.  Plus a lot of them are in use elsewhere.

The other issue is that a bunch of the modules just don't install with "carton install --deployment" ...  Here's the list of misses from the last run through :

! Couldn't find module or a distribution ExtUtils::MakeMaker (6.59)
! Bailing out the installation for Catalyst-Runtime-5.90015. Retry with --prompt or --force.
! Bailing out the installation for Catalyst-Plugin-AutoRestart-0.96. Retry with --prompt or --force.
! Couldn't find module or a distribution Carp (1.22)
! Bailing out the installation for Test-Warn-0.24. Retry with --prompt or --force.
! Bailing out the installation for Test-Most-0.25. Retry with --prompt or --force.
! Bailing out the installation for Test-Aggregate-0.364. Retry with --prompt or --force.
! Couldn't find module or a distribution Test::WWW::Mechanize::CGI (0)
! Bailing out the installation for REST-Utils-0.6. Retry with --prompt or --force.
! Couldn't find module or a distribution IO::Socket::SSL (0)
! Bailing out the installation for SOAP-Lite-0.714. Retry with --prompt or --force.
! Couldn't find module or a distribution B::Compiling (0)
! Bailing out the installation for Function-Parameters-0.04. Retry with --prompt or --force.

Any ideas ?

  • From: Tartu, EE
  • Registered: 2010-11-24

Re: Setting up a mirror on Ubuntu 12.04 - some issues

Try running it without --deployment (think I read some there were some issues with that going out of date).

  • Registered: 2012-08-28

Re: Setting up a mirror on Ubuntu 12.04 - some issues

Somewhat better.  Different in any case.

! Couldn't find module or a distribution Date::Parse (2.24)
! Bailing out the installation for Cache-2.04. Retry with --prompt or --force.
! Installing Net::SSLeay failed. See /home/downloads/.cpanm/build.log for details.
! Bailing out the installation for IO-Socket-SSL-1.76. Retry with --prompt or --force.
! Installing XML::Parser failed. See /home/downloads/.cpanm/build.log for details.
! Bailing out the installation for SOAP-Lite-0.714. Retry with --prompt or --force.
! Configure failed for XML-LibXML-2.0002. See /home/downloads/.cpanm/build.log for details.
! Bailing out the installation for HTML-HTML5-Parser-0.206. Retry with --prompt or --force.
Found conflicting versions for package 'inc::Module::Install'
  inc/Module/Install.pm (1.06)
  Module/Install.pm (0)
Found conflicting versions for package 'inc::Module::Install::DSL'
  inc/Module/Install/DSL.pm (1.06)
  Module/Install/DSL.pm (0)
! Configure failed for DBD-Pg-2.19.2. See /home/downloads/.cpanm/build.log for details.
! Installing XML::SAX::Expat failed. See /home/downloads/.cpanm/build.log for details.
! Bailing out the installation for XML-Simple-2.20. Retry with --prompt or --force.
! Installing Encode::Detect failed. See /home/downloads/.cpanm/build.log for details.

Still insists on installing its own Try::Tiny (and lots of others I've already installed via apt-get) though.

  • From: AZ, USA
  • Registered: 2012-08-20

Re: Setting up a mirror on Ubuntu 12.04 - some issues

Carton is designed to set up its own environment and ignore most system-installed packages; the notion here is that it means certain versions can be enforced. Unfortunately, CPAN also purges old packages after some period of time, which is what's resulting in the Date::Parse error (which is cascading to other failures), and related.

If you'd like to use system-installed packages, I'd recommend circumventing carton entirely. First, install cpanm (may be called cpanminus -- carton already uses it, so you may have it already). Then, from the musicbrainz-server checkout, run:

cpanm .

This will install all the dependencies listed in Makefile.PL, which is at least mostly kept in sync with carton.lock. For your DBD::Pg error, you should ensure you have postgresql installed -- I know in the past I've run into errors with it failing to install when postgresql wasn't. You should also read through the .cpanm/build.log file it mentions, to see if there are any error messages there that are informative. I'd also recommend either uninstalling as many system packages as you can first, or installing as many of the dependencies via the system as you can first -- be as close to one method or the other as you can muster, for maintainability's sake.

Once you've installed dependencies with cpanm, start the server with the same line listed, but without the 'carton exec --': 

plackup -Ilib -r

(this needs to be run from the musicbrainz-server directory, in case you try to do it from elsewhere). You may get errors in the vein of "couldn't find Foo/Bar.pm in @INC, INC contains..." -- this indicates something you need didn't get installed. This often happens with some optional dependencies, when you use the cpanm/Makefile.PL method; simply run: 

cpanm Foo::Bar

(obviously, replacing Foo::Bar with whatever you're actually missing) to correct each of these.

Just to jump the gun a bit, there's one other error you're likely to run into if you're running a replicated server (i.e. in DBDefs.pm you set REPLICATION_TYPE to RT_SLAVE): make sure, if you've set this up, that you have a READONLY database connection configured in DBDefs; otherwise you'll get a difficult-to-decipher "Validation failed for MusicBrainz::Server::Database with value undef" error.

And one last note: #musicbrainz-devel on freenode is where I and the other developers hang out, well, most of the time. If you have problems and would like responses in real time, that's probably the best place to go; http://webchat.freenode.net/ provides a web gateway, if that's more convenient.

ianmcorvidae's Website

5 Reply by halfwalker (edited by halfwalker 2012-08-29 00:01:37)

  • Registered: 2012-08-28

Re: Setting up a mirror on Ubuntu 12.04 - some issues

That's excellent, thanks very much for the time to spell it out.  My eventual aim is to have a system-installed-packages supported slave server, or as much as possible.  It's a thankless task of matching required dependencies with ubuntu packages - just grunt work really.  The drive for this is the headphones app and the issues around musicbrainz rate-limiting.

I'm assuming the READONLY definition in DBDefs.pm is the "sub DB_READ_ONLY { 1 }" statement ?  My mods are :

sub MB_SERVER_ROOT    { "/usr/local/musicbrainz-server" }
sub REPLICATION_TYPE { RT_SLAVE }
sub DB_STAGING_SERVER { 0 }
sub DB_READ_ONLY { 0 } (so replication updates work
sub DEVELOPMENT_SERVER { 0 }

This is on a working VM in virtualbox, and when all the perl deps are figured out from above, will be in a 12.04 system.

I've actually built a slick auto-installed downloading system - I use it for rebuilding my own downloader/mediaserver.  Boot a standard ubuntu 12.04 mini.iso with one minor tweak to use a specific preseed file, walk away for 30 minutes, and come back to a fully configured and integrated downloading and media serving system.  If anyone is interested, let me know - I'll release it soon.  It installs the following, fully configured and integrated :

File sharing via samba, afp, nfs, apple-timeshare
mdadm and/or zfs for media storage
usenet downloads via sabnzbd
torrent downloads via deluge or transmission
movie search via couchpotato
tv search via sickbeard
music search via headphones (hence the setup of a slave server)
photo/video gallery via gallery3
front-end via maraschino
statistics via munin
management via webmin
media streaming via plex, forked-daapd, subsonic, mini-dlna
virtualization via virtualbox
backups via crashplan
personal cloud via tonido
almost has openelec xbmc bootable via iscsi or aoe

Adding a pre-configured MB slave server here would be kinda cool.  One-stop shopping.

6 Reply by kurt1942 (edited by kurt1942 2013-04-13 12:51:28)

  • Registered: 2013-03-26

Re: Setting up a mirror on Ubuntu 12.04 - some issues

@halfwalker - I would be interested in your installer. I have been struggling to set up my own mirror on ubuntu and resorted to using the vm on my windows box. A click and go installer is an excellent idea :-)
Did this installer get released?