tag:hourglass.statuspage.io,2005:/historyHourglass Status - Incident History2023-04-05T03:43:40ZHourglasstag:hourglass.statuspage.io,2005:Incident/167552982023-04-05T03:04:52Z2023-04-05T03:04:52ZStatus page moved<p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>03:04</var> UTC</small><br><strong>Identified</strong> - This status page has been moved to https://status.chlod.net/.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>03:04</var> UTC</small><br><strong>Investigating</strong> - This status page has been moved to https://status.chlod.net/.</p>tag:hourglass.statuspage.io,2005:Incident/151582832022-12-14T11:23:25Z2022-12-14T11:23:25ZOngoing domain name transfer<p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>11:23</var> UTC</small><br><strong>Resolved</strong> - All good.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>08:49</var> UTC</small><br><strong>Monitoring</strong> - Domain has been transferred. Now verifying that everything works as expected.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>08:39</var> UTC</small><br><strong>Identified</strong> - The chlod.net domain name is currently being transferred to a different registry. Expect momentary DNS outages.</p>tag:hourglass.statuspage.io,2005:Incident/126507392022-10-28T23:01:58Z2022-10-28T23:01:58ZJTWC Archiver DNS issues<p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>23:01</var> UTC</small><br><strong>Resolved</strong> - Resolved with a temporary /etc/hosts workaround.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>22:53</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>22:52</var> UTC</small><br><strong>Investigating</strong> - Currently investigating domain name resolution issues with the JTWC website.</p>tag:hourglass.statuspage.io,2005:Incident/97519642022-04-10T14:11:15Z2022-04-10T14:11:15ZUpstream connectivity problems<p><small>Apr <var data-var='date'>10</var>, <var data-var='time'>14:11</var> UTC</small><br><strong>Resolved</strong> - The JTWC Archiver DNS record has been short-circuited locally for now (using /etc/hosts) due to a DNS caching issue. Unfortunately, the only system administrator responsible for this server will be signing off shortly, so this is the final resolution for now.</p><p><small>Apr <var data-var='date'>10</var>, <var data-var='time'>14:06</var> UTC</small><br><strong>Monitoring</strong> - The extreme slowdown is now being resolved. JTWC Archives are still being monitored for signs of life.</p><p><small>Apr <var data-var='date'>10</var>, <var data-var='time'>13:39</var> UTC</small><br><strong>Update</strong> - Discord bots are mostly unaffected, so they've been marked as operational once more. Other services (particularly parsers, scrapers, etc.) still refuse to work.</p><p><small>Apr <var data-var='date'>10</var>, <var data-var='time'>12:44</var> UTC</small><br><strong>Investigating</strong> - VPS 01 is currently experiencing connectivity issues that affect most outbound and inbound connections. This problem may be related to Hetzner networking as Hetzner websites are also unavailable. Information is still being gathered and we're trying to find ways to alleviate this problem in the meantime.</p>tag:hourglass.statuspage.io,2005:Incident/95817972022-03-28T01:56:01Z2022-03-28T01:56:01ZJTWC Website RRSIG Issues (upstream)<p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>01:56</var> UTC</small><br><strong>Resolved</strong> - The JTWC's DNSKEY records have been updated with a non-expired key. This upstream issue is now resolved.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>14:37</var> UTC</small><br><strong>Monitoring</strong> - The current CloudFront DNS A record has been verified and trusted manually, and an exception has been added to the system hosts file. The JTWC website will be continually monitored for DNSKEY compliance so that DNS records can be trusted automatically, at which point the record will be removed from the hosts file.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>11:27</var> UTC</small><br><strong>Identified</strong> - The JTWC Archiver has been experiencing archiving problems since 05:50 AM, 19 March (UTC). This is an upstream issue due to an RRSIG DNSKEY expiry for the `metoc.navy.mil` domain, leading to a distrust of the `metoc.navy.mil` domain for DNS providers. Since this is a security-related issue, no attempts will be made to circumvent this issue, and we will wait for a resolution on the United States Naval Meteorology & Oceanography Command's side. In the meantime, archive requests may be dropped.</p>tag:hourglass.statuspage.io,2005:Incident/91161192022-01-20T01:58:16Z2022-01-20T01:58:16ZJTWC Archiving failures<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>01:58</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>01:06</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>00:56</var> UTC</small><br><strong>Identified</strong> - The JTWC Archiver is experiencing continuous DNS resolution failures. We're investigating the issue and looking for a fix.</p>tag:hourglass.statuspage.io,2005:Incident/88419682021-12-14T14:13:18Z2021-12-14T14:13:18ZSMTP maintenance<p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>14:13</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>14:06</var> UTC</small><br><strong>Monitoring</strong> - The SMTP server is now available again.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>10:23</var> UTC</small><br><strong>Identified</strong> - The SMTP backend is currently under maintenance and will be unavailable for the time being.</p>tag:hourglass.statuspage.io,2005:Incident/85342692021-11-22T00:04:23Z2021-11-22T00:04:23ZAccidental SMTP Open Relay<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>00:04</var> UTC</small><br><strong>Resolved</strong> - The SMTP server is now being continuously monitored for an open-relay and is now operational.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>13:13</var> UTC</small><br><strong>Monitoring</strong> - Changes have been rolled back. An open relay is no longer detected. We will observe the server in the meantime to check if the problem reappears.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>12:50</var> UTC</small><br><strong>Identified</strong> - We're rolling back a recent firewall change. This might take a few moments.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>12:40</var> UTC</small><br><strong>Investigating</strong> - An accidental SMTP open relay has been detected. The postfix server responsible for mail has been temporarily disabled.</p>tag:hourglass.statuspage.io,2005:Incident/82253632021-10-17T09:25:25Z2021-10-17T09:25:25ZPlanned Power Interruption<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>09:25</var> UTC</small><br><strong>Resolved</strong> - The JTWC Archiver was shifted to the main system (no proxies involved) prior to the interruption. According to the execution history, the main system was able to perform all archiving tasks without issue.</p><p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>06:03</var> UTC</small><br><strong>Identified</strong> - A planned power interruption at the hosting service for the JTWC Archiver's proxy will take the archiver down for 11 hours (16 October, 23:00 UTC to 17 October, 10:00 UTC). This may last shorter or longer than expected.</p>tag:hourglass.statuspage.io,2005:Incident/81119432021-10-02T01:29:37Z2021-10-02T01:29:37ZProxy server unreachable<p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>01:29</var> UTC</small><br><strong>Resolved</strong> - The problem has been resolved and the JTWC Archiver has been running properly for the past few days with an error rate of 0.6%.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>14:41</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented, which forces connections to go through the JTWC's IPv4 IP instead of its IPv6 IP (which is unreachable). We will continue to monitor the service for any problems or interruptions.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>14:06</var> UTC</small><br><strong>Update</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>13:55</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>13:55</var> UTC</small><br><strong>Identified</strong> - The proxy server responsible for tossing requests from the VPS to a residential IP address is currently unavailable. This will cause attempts at archiving JTWC notices fail.</p>tag:hourglass.statuspage.io,2005:Incident/76762542021-08-14T09:07:22Z2021-08-14T09:07:22ZJTWC Website Inaccessible<p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>09:07</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>08:02</var> UTC</small><br><strong>Update</strong> - Fix is operational as of Fri 13 Aug 2021 08:00:00 UTC.</p><p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>07:19</var> UTC</small><br><strong>Monitoring</strong> - The fix has been implemented, and the archiver will be observed for a 24-hour period beginning Fri 13 Aug 2021 07:20:00 UTC. It should continue its per 10-minute archive cycle per usual.</p><p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>06:40</var> UTC</small><br><strong>Update</strong> - The problem has been identified as an IP/User Agent block. A fix is currently being implemented, and additional restrictions are being added to prevent overloading the JTWC servers in the future.<br /><br />If the archiver is blocked again after applying the relevant fixes and all attempts at contacting the JTWC fail, the archiver will be permanently decommissioned.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>16:54</var> UTC</small><br><strong>Update</strong> - The JTWC is blocking all connections from the Hourglass VPS. We are currently looking for ways to resolve this problem.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>00:20</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>00:19</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>00:18</var> UTC</small><br><strong>Identified</strong> - Live updates are not being made to the archiver. Since this is an upstream issue, we cannot do anything on our end to fix this problem.<br /><br />Please wait for the JTWC website (https://www.metoc.navy.mil/jtwc/jtwc.html) to return to normal functionality.</p>tag:hourglass.statuspage.io,2005:Incident/75817672021-07-23T04:16:06Z2021-07-23T04:16:22ZTotal loss of connectivity<p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>04:16</var> UTC</small><br><strong>Resolved</strong> - VPS 01 lost IPv4 connectivity at around 22 July 2021, 16:30. This caused an 11-hour outage which brought down almost all outgoing and incoming connections, including multiple services which relied on the internet.</p>