I’m thinking the title of this post may one day be a best-selling e-book written by myself. The word “overnight” is a slight exaggeration, however since the second week of August, leading up to the present point in time, I have been working tirelessly (yes even through the night and over a weekend!), to migrate a web design/ hosting company’s complete hosting solution to local premises, from the over-hyped cloud.
Now, this all sounds very exciting, and it is (for me), however the complexity of moving from a hosted server environment (thanks to lcn.com) to a more localised, manageable system is enormous. Imagine turning up to a new job, and on day one the boss says, here’s your predecessor’s system, there’s no documentation on how to do your job just log in and figure out what you’re going to be doing without much help from those around you as they’re all too busy; well that’s a slightly over-hyped version of the situation here. Inheriting a system without documentation is one thing, gradually by pooling all the resources you can find you can map it out into your own terms and understanding – and can gain a good basic overview, then with more time (if available) you learn all the in depth aspects of the system and can sometimes if lucky, rewrite certain elements to your own language and document that – this is in essence reverse engineering, and a good skill to have as a system admin.
The second problem, once system knowledge is gained is, how to go about migrating from a hosted environment to one you host yourself. This sounds easy – in short it’s not! With over 60 customer’s websites and emails hosted, moving this is inevitably going to cause some disruption and downtime, this can be managed with good customer service, however what happens if certain things don’t work properly on the new system, or after a reboot not all services come back, or you don’t know the password for the user that runs a specific task? Yes, I have experienced all these and more. In fact, this whole challenge has stretched my skills and knowledge to the limit, and I’m sure once the new systems are fully live and more customers come on-board, I will look back and be grateful for having this experience.
Migration in this case hasn’t been awful, because, even though the servers are dedicated server environments, the actual hosting servers are virtualised within those, and with a little Linux/ virtualisation magic (and a weekend window of 2 hours), images were cloned and copied to the new servers. Cloning did mean taking all systems down temporarily, but I was impressed with the speed of cloning 100Gb images (under 45mins) – and with 100Mbps Fibre at my end the download of the clone wasn’t too bad either.
So, now I have the clones what do I do with them – well assuming my local servers are configured correctly to host the virtual images I can simply turn them on, change the IP addresses, do my usual Firewall magic (god bless WatchGuard!), and away we go… Could it really be that simple, well… Yes, surprisingly. In my head I thought “turn it on, change the ip addresses, reboot, test” – And wuite simply this was the case. So now I have a virtualised hosting environment, running on much more powerful equipment of which I have full control. Sorted.
Well, almost… What about the websites themselves, I may have the data, and even the emails we host, but what about the pointers to those sites, where do nameservers point, who has control…Bugger! – The first panic/ minor snag was realising we dont manage the domains for all the customers, and after investigation, we only manage around half (usually where they have .co.uk and .com and we set up one but they did the other!) – OK not to worry, should we just do it in one hit and modify the nameservers, well we could, but I don’t like the risk, but I do like control, so we decided to manually switch one domain at a time, allownig to test each one individually and ensure customer satisfaction is still at the highest levels. – This is a slow process, especially where back-end DBs and emails are still being modified all the time, as I have a new system not being updated whilst the old is (and vice-versa) so a little manual intervention is required to copy down data from the “old” to the “new”, but again this gives me control so it’s all good.
Now, it’s just a case of going through the list of domains (all 132) one by one and modifying nameservers, and informing customers to do the same where they have the control. I like the personal touch that’s possible with this many customers, but dread what we would have to plan if we did this again with 10x the numbers (which is always possible)…
Thanks for reading my longest post ever, for more information on this ongoing project and the tools and skills required please feel free to contact me at: firstname.lastname@example.org