Latency on Level3 (Resolved)
  • Priority - Critical
  • Affecting Other - Level3
  • Level3 is experiencing massive latency throughout the US on their backbone.  We are continuing to monitor the situation and may disable our L3 link later in the day if its not corrected.

  • Date - 02/02/2017 07:16
  • Last Updated - 02/04/2017 06:46
Scheduled Vendor Network Maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Cogent Uplink
  • Webby Enterprises LLC has been notified that one of our upstreams, Cogent Communications, will be performing maintenance on routers in the near future.  Webby Enterprises LLC will be de-activating the circuit in question to avoid issues within our network.

    Start time: January 19, 2017 12:00 AM ET
    End time: January 19, 2017 06:00 AM ET

    * Webby Enterprises LLC will de-activate the circut in question at 8pm ET on January 18, 2017 and reactivate it at 7AM ET on January 19, 2017 after maintenance has been concluded.  While the circuit is deactivated on the 18th and reactivated on the 19th, you may experience some routing issues briefly.

  • Date - 01/06/2017 06:19 - 01/19/2017 03:10
  • Last Updated - 01/19/2017 05:59
Server Reboots (Resolved)
  • Priority - Medium
  • Affecting Other - ALL SYSTEMS
  • Webby Enterprises LLC will be rebooting Shared, Managed VPS, Managed Dedicated and VPS Nodes one a a time on a rolling basis Saturday and Sunday (January 14 and 15) to apply patches.

    Downtime should be minimal at 5-6 minutes for the server reboot.

    If you have any questions, please contact our NOC.

  • Date - 01/13/2017 14:56 - 01/14/2017 09:00
  • Last Updated - 01/16/2017 10:38
Alvin Outage (Resolved)
  • Priority - Critical
  • Affecting Server - alvin.genwebserver.com
  • We are aware of a issue affecting Alvin at the moment.  NOC staff is currently investigating.

    Update: 11/13/2016 12:50PM ET
    It appears Alvin went to read-only for a unknown reason at approxiately 12:35pm.  A FSCK was completed and a reboot is in progress.  If needed, CDP backup from approxiamtely 1:30am ET is available and will be used for data recovery.

    Update: 11/13/2016 1:00PM ET
    Root cause has been determiend and it has been determined we will need to utilize the CDP backup from 1:30am ET this morning.  This process has started and we will provide updates as it progresses.

    Update: 11/13/2016 1:20PM ET
    CDP restore has been kicked off and is actively restoring data as of 1:20pm ET.

    Update: 11/13/2016 1:50PM ET
    CDP restore continues and is currently at 151GB of 232GB restored.  ETA is roughly 25 minutes however that can change based on the data transfer rates.

    Update: 11/13/2016 1:55PM ET
    Alvin is back up and responding.

  • Date - 11/13/2016 12:25
  • Last Updated - 11/13/2016 13:58
Theodore (mySQL) (Resolved)
  • Priority - Critical
  • Affecting Server - theodore.genwebserver.com
  • At 11:30am ET, Webby Enterprises LLC Network Operations was alerted to mySQL on Theodore in crash recovery.

    We are currnetly investigating and workign to restore mySQL access on Theodore.

  • Date - 06/21/2016 11:44
  • Last Updated - 06/21/2016 11:55
Simon (Failed RAID Card) (Resolved)
  • Priority - Critical
  • Affecting Server - simon.genwebserver.com
  • At approximately 5:02am ET, Webby Enterprises LLC was alerted to Simon in a hard down state. Upon investigating, Simon's RAID card appears to have malfunctioned causing the RAID to be lost.

    We are dispatching a service technician to the data center in Saint Louis to replace the RAID card. 

    At this time there is no ETA, however our SLA will be in affect.

  • Date - 06/21/2016 05:02 - 06/21/2016 10:25
  • Last Updated - 06/21/2016 10:28
Simon (Resolved)
  • Priority - Critical
  • Affecting Server - simon.genwebserver.com
  • At 4:08pm ET, Webby Enterprises LLC was alerted to Simon in a Hard Down state while CPanel Engineers were investigating a issue.  A server reboot and investigation is in progress.

  • Date - 05/27/2016 16:11
  • Last Updated - 05/27/2016 19:33
Simon (mySQL) (Resolved)
  • Priority - Critical
  • Affecting Server - simon.genwebserver.com
  • We are continuing to investigate a mySQL issue with Simon.  Our technicians have engaged with cPanel at this time.

    UPDATE 9:00am ET
    InnoDB has crashed and we are working on crash recovery at this time.

    UPDATE 3:30pm ET
    mySQL has again crashed due to corrutped InnoDB tables.  We are taking the mySQL server offline into crash recovery in an attempt to fix those tables.

    UPDATE 5:00pm ET
    InnoDB based databases that are corrupted will need restored from a offsite backup.  If your database is affected and you would like us to restore your offsite backup, please open a support ticket.

  • Date - 04/27/2016 08:06
  • Last Updated - 05/08/2016 11:00
Simon:: FSCK (Resolved)
  • Priority - Critical
  • Affecting Server - simon.genwebserver.com
  • Simon stopped processing requests.  A reboot was issued and it isundergoing a FSCK on /dev/sda3

    Update 1/7/16 10:00pm ET - Simon has been rebooted, fsck'd and is responding to requests.

  • Date - 01/07/2016 21:39 - 01/07/2016 21:58
  • Last Updated - 01/07/2016 21:58
Simon (Resolved)
  • Priority - Critical
  • Affecting Server - simon.genwebserver.com
  • Webby Enterprises LLC is investigating Simon who went down at 9:50pm ET and will not boot at this time.

    -----------------
    Update 10:50pm ET - At this time, we continue to attempt to bring Simon back up.  We will be making the call shortly whether to start restoring the 1am ET backup to another machine or not.

    -----------------
    Update 11:00pm ET - Simon is back online and we will be reviewing logs to determine what happened.  We will most likely be ordering a new SSD based server to switch to for Simon to rule out ALL hardware.

  • Date - 11/02/2015 22:04
  • Last Updated - 11/02/2015 22:59
Simon Outage (Resolved)
  • Priority - Critical
  • Affecting Server - simon.genwebserver.com
  • Webby Enterprises LLC Network Operations is investigating an outage with Simon

    Update 10/15/2015 05:18am ET
    Bug with the Kernel appears to have caused a kernel panic and IRQ messages.  We have deployed a fix in the kernel boot parameters and has been rebooted is now bringing services online.

    Update 10/15/2015 05:35am ET
    Simon has been brought back online and appears to be ok.  Network Operations will continue to monitor for further issues.

  • Date - 10/15/2015 05:00
  • Last Updated - 10/15/2015 05:38
Credit Card Processing (Resolved)
  • Priority - Critical
  • Affecting Other - Credit Card Payment System
  • Webby Enterprises LLC is aware of a issue affecting online credit card payments to us. We are working with vendors to resolve it at this time.

  • Date - 08/05/2015 12:00
  • Last Updated - 08/05/2015 17:02
Simon (Reboot) (Resolved)
  • Priority - Critical
  • Affecting Server - simon.genwebserver.com
  • Simon is being rebooted to apply a patch to fix the random kernel panics

    As of 5:05pm ET, Simon has been rebooted and is back online and functional.

  • Date - 07/13/2015 17:00 - 07/13/2015 17:05
  • Last Updated - 07/13/2015 17:05
Simon (Not Responding) (Resolved)
  • Priority - Critical
  • Affecting Server - simon.genwebserver.com
  • We are aware of Simon not responding to requests from the internet and are currently investigating

  • Date - 07/09/2015 10:08 - 07/09/2015 10:20
  • Last Updated - 07/09/2015 11:20
Theodore (Web Outage) (Resolved)
  • Priority - Critical
  • Affecting Server - theodore.genwebserver.com
  • Theodore is currently experiencing a high load related to the web service

  • Date - 06/24/2015 15:28 - 06/24/2015 21:00
  • Last Updated - 06/25/2015 12:01
Simon Rebooting (Resolved)
  • Priority - Critical
  • Affecting Server - simon.genwebserver.com
  • Simon is currently rebooting for an emergency kernel repair

  • Date - 06/20/2015 22:25 - 06/20/2015 22:39
  • Last Updated - 06/20/2015 22:39
Outbound Email (Blacklisted) (Resolved)
  • Priority - Critical
  • Affecting Other - All Shared/Managed Customers
  • Webby Enterprises LLC is aware of a current blacklisting causing some mail delivery issues, specifically to Hotmail.com/Outlook.com/Live.com<br /><br />We have corrected the problem and are working with the blacklists to remove ourselves from the Blacklist services.

    Update 1/17/2015 08:50AM ET
    Webby Enterprises LLC has turned up an additional Spam Compliance Cluster member to alleviate the strain on the IP thats blacklisted.  Mail will now be flowing through 2 different mail gateway appliances for outbound processing so mail may return will be returning based on what gateway your mail gets sent out.  We continue to work with Hotmail to delist the IP

    Update 1/18/2015 07:00AM ET
    As of 1/17/2015 19:44:57 our monitoring system reports that our Reputation with SenderBase on 1 IP has returned to the "Good" level.  We are still awaiting a response from Microsoft on the Hotmail/Outlook/Live blacklisting.

    Update 1/18/2015 02:00PM ET
    As of now, Microsoft is reporting all outgoing mail cluster IPs free and clear

  • Date - 01/16/2015 15:12 - 01/18/2015 14:00
  • Last Updated - 01/18/2015 14:51
Simon Outage (Resolved)
  • Priority - Critical
  • Affecting Server - simon.genwebserver.com
  • Maintenance Type:

    • Server

    Maintenance Severity:

    • Critical

    Duration:

    • We will update as details become available.

    Detailed Description:

    • simon.genwebserver.com is not responding when queried. This may be caused by an unexpected restart leading to a file system check, a hardware issue, or network error. Our system administrators are already working to resolve this issue.
    Customers or services affected:
    • simon.genwebserver.com

     Updates:

    •  5:43AM EST simon.genwebserver.com has been rebooted and appears to be back online and functional.  Engineering will now work to determine the exact cause of outage

  • Date - 12/23/2014 23:00 - 12/24/2014 05:45
  • Last Updated - 12/24/2014 05:12
Data Center Access/Security (Resolved)
  • Priority - High
  • Affecting Other - Data Center
  • In anticipation of the Grand Jury announcement for the Furgeson, MO case, Webby Enterprises LLC along with our Data Center partners are invoking Elevated Security Protocols.

    At this time, only Badged Access will be permitted through the main entrance (Front Lobby) of the Building.  No Entry through any other normally used permiter doors will be permitted.

    All deliveries must be made through the Loading Dock with Customer Escort.

    All Vendors must be escorted and are subject to search along with equipment/belongings.

    Visitors are RED Badge/Escort Only

    This protocol will remain in effect until the current situation changes or an All Clear is given.  Webby Enterprises LLC will continue to update our customers as updates become available.

  • Date - 11/24/2014 19:25
  • Last Updated - 12/01/2014 11:03
Power Maintenance - 11/28/2014 (Resolved)
  • Priority - High
  • Affecting Other - Power Infrastructure
  • Digital Realty Trust will be performing maintenance within the Webby Enterprises LLC infrastructure on November 28, 2014 from 8:30AM to 4:30PM

    During this time, breakers may inadvertently trip while being adjusted.  If a breaker would inadvertently trip during the adjustments, LOAD will transfer to generator power.  Load will remain on generator until all adjustments are completed and a coordinated retransfer can be performed.

  • Date - 11/28/2014 08:30 - 01/16/2015 19:05
  • Last Updated - 11/18/2014 09:46
Data Center Power Feed (Resolved)
  • Priority - Medium
  • Affecting Other - Data Center
  • Ameren Missouri has contacted our Data Center Property Management team to let us know of a potential issue with our Medium Voltage feeds. Due to a duct bank collapse downtown that was related to a sewer line dig the three feeds are at risk - two of the three feeds are exposed and still intact, but because of the added exposure there is an elevated risk of impact to the site. 

    Contractors are evaluating what needs to be done to shore up the area. It is unknown at this time what the timeline is for repairs. 

    The Data Center Property Management remains in contact with Ameren and is monitoring the situation closely. The site operations team has proactively checked fuel levels for generators and performed a prestart checklist to insure generators are available if needed. All Change Management for Generator PM scheduled for this week has been postponed until we receive the all clear from Ameren. We will notify customers of any change in the situation or if there are any updates.

    -------------------------------------------
    11/3/2014 4:20 PM Eastern Update
    As an update to the earlier Medium Voltage Service feed issue identified by Ameren Missouri and due to their planned corrective maintenance; the issue now involves all feeds that impact the entire site. 

    Ameren reported that their duct bank on 11th Street between Pine & Chestnut has partially collapsed, exposing two feeders but there is no apparent damage. Ameren is going to take the reserve feeder to the Data Center out of service this afternoon while leaving our main feeders online. Once out of service, Ameren plans to carefully remove the remaining damaged duct bank and inspect the cables. Following that, Ameren plans to make provisions to protect and support the duct bank so that MSD can continue excavation work in the area. If there is not significant damage to the existing cables, all feeders will be re-energized at this time. Once MSD work is complete later this week, Ameren will install conduit sleeves around each of the cables and conduits then encase the conduit bank in new concrete and backfill the excavation.  

    Data Center Property Management will continue to monitor the situation closely. The site operations team has proactively checked fuel levels for generators and performed a prestart checklist to insure generators are available if needed. All Change Management for Generator PM scheduled for this week has been postponed until we receive the all clear from Ameren. The next scheduled update from Ameren is the morning of Tuesday, November 4. We will notify customers of any change in the situation.

    -------------------------------------------
    11/4/2014 12:30 PM Eastern Update
    All circuits associated with the 11th street collapse have been secured and released to be restored. Ameren crews have sleeved the cables and supported the conduits while MSD completes their work. The rebuilding of the duct bank will be completed once the contractor has finished their work. Ameren does not plan to take POP #53,#55 and #57 circuits out of service.  

    Two out of the three Spot Network (#2 and #3) feeders are back on line supporting our Community Service 1 (CS 1) customers. However, Ameren has found a blown heat shrink splice near 14th & Olive on the remaining Spot Network Pop-#11. Crews will work to replace a section of cable and make new splices.  

    All Change managements for annual Generator PM’s are back on schedule. 

    We will continue to provide updates as addition information is received from Ameren.

    -------------------------------------------
    11/5/2014 4:30 PM Eastern Update
    Ameren’s crews are still working on the Spot Network Pop-11 feeder. At 3pm CST yesterday, Ameren was able to pull in a new section of cable and crews are working on splicing the new cable in. Crews will continue to work until complete.  

    The collapsed duct bank is in the process of being back filled. 

    We will continue to provide additional updates as they are received from Ameren.

    -------------------------------------------
    11/6/2014 8:30 AM Eastern Update
    Ameren has reported the POP 11 feeder on the Spot Network supporting our Community Service 1 customers is back on line. 

    Ameren will backfill and repair the concrete duct bank supporting POP 53,55 and 57 today. 

    We will continue to provide additional updates as they are received from Ameren.

  • Date - 11/03/2014 13:41
  • Last Updated - 11/14/2014 06:15
Simon (Resolved)
  • Priority - Critical
  • Affecting Server - simon.genwebserver.com
  • We are currently investigating a outage on Simon
    ----------------------------
    11/14/2014 7:15 AM Eastern

    Simon appears to be back up.  We will continue to investigate the root cause

  • Date - 11/14/2014 05:50 - 11/14/2014 07:15
  • Last Updated - 11/14/2014 06:15
Simon (RAID Issue) (Resolved)
  • Priority - Critical
  • Affecting Server - simon.genwebserver.com
  • We are currently experiencing RAID issues on Simon.  We are currently working on a RAID Rebuild.  Additional updates are/will be availabel on the Facebook group
    -------------------
    As of Noon, Simon is back up with the RAID back to operational state and Sept 30 backup restored.

  • Date - 10/01/2014 05:00 - 10/01/2014 12:02
  • Last Updated - 10/01/2014 12:05

Powered by WHMCompleteSolution