Our planned seamless DNS move today was not quite as seamless as planned.
A routing issue caused a corruption in refresh times, and around 200 domains (including our own) ceased to resolve for around 4 hours.
We placed an emergency resolution ticket with our core network team and the issue was quickly resolved. We apologise for any inconvenience.
An issue with our database replication led to a brief outage of DNS resolution this morning. Our systems picked up on this and the issue was resolved at 10.00hrs BST.
If you are still experiencing problems with DNS records, please email email@example.com
As long-standing customers will be aware, March and April are our busiest times of the year as we have clients with underspend who need completion before the new tax year weighed against those who spent their money last November and will be chomping at the bit as soon as the new tax year starts and - basically - everybody wants everything yesterday, if not sooner.
The crew are currently working silly-daft hours and although he keeps offering to come in and lend a hand, we've banned Martyn from the offices as he's recovering from a very painful operation.
THEREFORE there may be some delay in responding to technical queries of a non-urgent nature, or repetitive issues that we have previously fixed, and some customers (you know who you are!) have promptly 'un-fixed'.
Essentially, the more time we have to spend dealing with your major issue of not being able to tune into iPlayeron your iPad (neither of which are our responsibility) the less time we have to do 'real work', like answering the phone to invaluable PPI parasites...
We will deal with all setup and modification requests on a 'first come, first actioned' basis BUT WILL ADDITIONALLY WEIGH THE PRIORITY to take into account the severity of the request, and no, iPlayer and iPad don't come anywhere near the top of our priorities, in fact, they're not even on the list.
If we tell you it isn't a problem here, and your 'technical support IT provider' claims otherwise, you need to either change them, or us.
Thank you for your understanding.
Head of Technical Operations
Due to an unusually high number of new orders (customers with underspend in their budget which needs spendi9ng before the new Tax Year) together with new orders pending to be provisioned as soon as the new Tax Year starts - and finally, our own 'year end', we've taken the decision to close the Sheffield office for a couple of weeks until we catch up and clear our list of 'things toi do'.
Therefore the Sheffield office will be closed until Tuesday 10th April.
During this time support may be obtained by either leaving a voicemail on our main number (0114 258 5550) or by email.
Emergency support can be obtained by calling the Data Centre Duty Manager on 0114 255 5959
(Please, only for emergencies).
Below is a history based on the most recent 3 updates we have received:
BT last night, advised they expected the water to have been completely
pumped out from the exchange by approximately 00:00 with power being
restored to the exchange in logical phases, the coupling of the
generators is estimated to have been completed by 00:30 as the
preparatory work for this activity has been conducted in parallel to
the pumping activity. Power will be restored to the individual area's
in priority order with blue light services being the initial focus.
BT, at 06:30 today confirmed they have started restoring power with the
majority of traffic still remaining down, the site is being restored in
a pre-planned methodical pattern with priority blue light services
identified earlier as main focus.
BT as of 09:00 today, advised that 57 of the affected 60 racks have
power restored and are in the process of restoring the remaining 3 with
an expected completion time of 1 hour. BT also advised that although
power will be restored this will not guarantee service restoration as
kit and configuration will need to be checked after power up. BT has
also confirmed that all possible re-routes have been completed. Any
circuits remaining that have not been rerouted are not able to be.
This issue has affected:
- All PSTN lines terminating at Paddington and some nearby exchanges.
- All IP Stream ADSL circuits terminating at Paddington.
- IP Stream ASDL circuits at other exchanges nationwide, where the ATM
backhaul path is to Paddington.
- Around 35000 ADSL Datastream customers nationwide due to
multiple backhaul circuits having been affected.
We will provide further updates as we receive them.