<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2025-09-08 Mon 17:41 -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="generator" content="Org Mode">
</head>
<body>
<div style="overflow:auto;font-family:"Arial";font-size:10pt;line-height:11pt;overflow:auto;margin:0 auto;" id="content">
<p style="text-decoration:none;margin-bottom:0px;margin-top:10px;line-height:11pt;font-size:10pt;font-family:"Arial";">
Sorry for the giant spam of emails.<br>
I just restarted flywheel and it caught up with all the scans.<br>
</p>
<p style="text-decoration:none;margin-bottom:0px;margin-top:10px;line-height:11pt;font-size:10pt;font-family:"Arial";">
This is separate to whatever UPMC/Pitt cross talk network hangup prevented messages from sending last.<br>
</p>
<hr>
<p style="text-decoration:none;margin-bottom:0px;margin-top:10px;line-height:11pt;font-size:10pt;font-family:"Arial";">
For posterity (find this message in the <a style="color:#0071c5;" href="https://list.pitt.edu/pipermail/flywheelgearlist/">
list archive</a>)<br>
</p>
<div style="overflow:auto;font-family:"Arial";font-size:10pt;line-height:11pt;" id="outline-container-org0000000">
<h2 style="margin-top:20px;margin-bottom:20px;font-style:italic;color:#0071c5;font-size:13pt;font-family:"Arial";" id="org0000000">
Postmortem</h2>
<div style="overflow:auto;font-family:"Arial";font-size:10pt;line-height:11pt;" id="text-org0000000">
<p style="text-decoration:none;margin-bottom:0px;margin-top:10px;line-height:11pt;font-size:10pt;font-family:"Arial";">
Zeus (one of the computers in the cold room next to prisma 1) had some unplanned down time (power issue?). It hosts the virtual machines that run flywheel. It all came back up Friday but a flywheel bug currently requires manual intervention (via
<a style="color:#0071c5;" href="https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNPACore%2Fhpc-env%2Fblob%2Fmaster%2Fflywheel%2Freset-for-pending-jobs.bash&data=05%7C02%7Cflywheelgearlist%40list.pitt.edu%7Cc9b7707df1074bf5ffc808ddef206e1f%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C638929644736768392%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Zt7CI3yKEc9BV1Kx9B8eSi8ShSi0Qtw8UMMEmTecFP4%3D&reserved=0" originalsrc="https://github.com/NPACore/hpc-env/blob/master/flywheel/reset-for-pending-jobs.bash">
<code style="font-size:10pt;font-family:monospace;background:#f9f9f9;">/raidzeus/src/hpc-env/flywheel/reset-for-pending-jobs.bash</code></a>) to get jobs out of a "pending" state. I tweaked cron on zeus so this hopefully wont happen again.<br>
</p>
</div>
<div style="overflow:auto;font-family:"Arial";font-size:10pt;line-height:11pt;" id="outline-container-org0000001">
<h3 style="margin-bottom:0px;text-decoration:underline;color:#0071c5;font-size:12pt;font-family:"Arial";" id="org0000001">
uptime</h3>
<div style="overflow:auto;font-family:"Arial";font-size:10pt;line-height:11pt;" id="text-org0000001">
<p style="text-decoration:none;margin-bottom:0px;margin-top:10px;line-height:11pt;font-size:10pt;font-family:"Arial";">
Gyrus, zeus, and cortex were all part of the same downtime? But <code style="font-size:10pt;font-family:monospace;background:#f9f9f9;">
cerebro2</code> stayed up.<br>
</p>
<div style="margin-left:-.5em;margin-top:10px;overflow:auto;font-family:"Arial";font-size:10pt;line-height:11pt;">
<pre style="padding-left:2em;padding:1em;border:solid #ccc 2px;line-height:12pt;color:#abb2bf;background-color:#282c34;margin:0px;font-size:9pt;font-family:monospace;"><span style="color: #00d3d0;">(</span> <span style="color: #ff7f86;">for</span> host<span style="color: #ff7f86;"> in</span> mrrc-<span style="color: #ff5f59;">{</span>zeus,cerebro2,cortex<span style="color: #ff5f59;">}</span>; <span style="color: #ff7f86;">do</span>
  ssh $<span style="color: #6ae4b9;">host</span> -- echo -e <span style="color: #00d3d0;">"${host/mrrc-/} \$(</span><span style="color: #feacd0;">uptime -s</span><span style="color: #00d3d0;">)"</span>
<span style="color: #ff7f86;">done</span>
<span style="color: #6ae4b9;">SSHPASS</span>=$<span style="color: #ff5f59;">(</span><span style="color: #feacd0;">pass gyrus2</span><span style="color: #ff5f59;">)</span> sshpass -e ssh mrrc-gyrus2 -- echo -e <span style="color: #00d3d0;">"gyrus2 \$(</span><span style="color: #feacd0;">uptime -s</span><span style="color: #00d3d0;">)"</span> <span style="color: #00d3d0;">)</span> |
sort -k2 -t<span style="color: #00d3d0;">' '</span>
</pre>
</div>
<table style="font-family:"Arial";font-size:10pt;margin-top:0px;line-height:10pt;border-collapse:collapse;">
<colgroup><col class="org-left"><col class="org-right"><col class="org-right"></colgroup>
<thead>
<tr>
<th style="border:1px solid white;background-color:#0071c5;color:white;padding-left:10px;padding-right:10px;" scope="col">
host</th>
<th style="border:1px solid white;background-color:#0071c5;color:white;padding-left:10px;padding-right:10px;" scope="col">
uptime_date</th>
<th style="border:1px solid white;background-color:#0071c5;color:white;padding-left:10px;padding-right:10px;" scope="col">
uptime_time</th>
</tr>
</thead>
<tbody>
<tr>
<td style="font-family:"Arial";font-size:10pt;margin-top:0px;padding-left:10px;padding-right:10px;background-color:#f9f9f9;border:1px solid white;text-align:left;">
cerebro2</td>
<td style="font-family:"Arial";font-size:10pt;margin-top:0px;padding-left:10px;padding-right:10px;background-color:#f9f9f9;border:1px solid white;text-align:right;">
2025-08-01</td>
<td style="font-family:"Arial";font-size:10pt;margin-top:0px;padding-left:10px;padding-right:10px;background-color:#f9f9f9;border:1px solid white;text-align:right;">
04:09:44</td>
</tr>
<tr>
<td style="font-family:"Arial";font-size:10pt;margin-top:0px;padding-left:10px;padding-right:10px;background-color:#f9f9f9;border:1px solid white;text-align:left;">
zeus</td>
<td style="font-family:"Arial";font-size:10pt;margin-top:0px;padding-left:10px;padding-right:10px;background-color:#f9f9f9;border:1px solid white;text-align:right;">
2025-09-05</td>
<td style="font-family:"Arial";font-size:10pt;margin-top:0px;padding-left:10px;padding-right:10px;background-color:#f9f9f9;border:1px solid white;text-align:right;">
04:14:02</td>
</tr>
<tr>
<td style="font-family:"Arial";font-size:10pt;margin-top:0px;padding-left:10px;padding-right:10px;background-color:#f9f9f9;border:1px solid white;text-align:left;">
gyrus2</td>
<td style="font-family:"Arial";font-size:10pt;margin-top:0px;padding-left:10px;padding-right:10px;background-color:#f9f9f9;border:1px solid white;text-align:right;">
2025-09-05</td>
<td style="font-family:"Arial";font-size:10pt;margin-top:0px;padding-left:10px;padding-right:10px;background-color:#f9f9f9;border:1px solid white;text-align:right;">
04:15:14</td>
</tr>
<tr>
<td style="font-family:"Arial";font-size:10pt;margin-top:0px;padding-left:10px;padding-right:10px;background-color:#f9f9f9;border:1px solid white;text-align:left;">
cortex</td>
<td style="font-family:"Arial";font-size:10pt;margin-top:0px;padding-left:10px;padding-right:10px;background-color:#f9f9f9;border:1px solid white;text-align:right;">
2025-09-05</td>
<td style="font-family:"Arial";font-size:10pt;margin-top:0px;padding-left:10px;padding-right:10px;background-color:#f9f9f9;border:1px solid white;text-align:right;">
08:11:57</td>
</tr>
</tbody>
</table>
</div>
</div>
<div style="overflow:auto;font-family:"Arial";font-size:10pt;line-height:11pt;" id="outline-container-org0000002">
<h3 style="margin-bottom:0px;text-decoration:underline;color:#0071c5;font-size:12pt;font-family:"Arial";" id="org0000002">
automated fix</h3>
<div style="overflow:auto;font-family:"Arial";font-size:10pt;line-height:11pt;" id="text-org0000002">
<p style="text-decoration:none;margin-bottom:0px;margin-top:10px;line-height:11pt;font-size:10pt;font-family:"Arial";">
<code style="font-size:10pt;font-family:monospace;background:#f9f9f9;">cron</code> should run what's needed a few minutes after the systems come back online. ðŸ¤ž 10 minutes is enough time to give the system before restarting FW.<br>
</p>
<div style="margin-left:-.5em;margin-top:10px;overflow:auto;font-family:"Arial";font-size:10pt;line-height:11pt;">
<pre style="padding-left:2em;padding:1em;border:solid #ccc 2px;line-height:12pt;color:#abb2bf;background-color:#282c34;margin:0px;font-size:9pt;font-family:monospace;">ssh foranw@mrrc-zeus crontab -l |grep pending-jobs
</pre>
</div>
<pre style="padding-left:2em;padding:1em;border:solid #ccc 2px;line-height:12pt;color:#abb2bf;background-color:#282c34;margin:0px;font-size:9pt;font-family:monospace;">
@reboot sleep 600 && /raidzeus/src/hpc-env/flywheel/reset-for-pending-jobs.bash
</pre>
</div>
</div>
</div>
</div>
</body>
</html>