I find that every project I work on I tend to steer the code in a different direction. Often this is a response to the project. With Python, I have this flexibility – at times I focus on functions, other times I’ll give data the bulk of my attention.
I used to write a lot of small Bash and Python interactions, but more and more my experience is trending toward larger projects that involve constructing an entire back-end using these languages. Either way – over the years I have written some neat code (I think).
I encountered a service with an inconvenient combination of problems: it was slow to get all data, process it, and submit responses back incrementally. I couldn’t batch the problem, but this all left the interaction vulnerable to the session token timing out.
The process to resolve this involved implementing Python
dataclasses to manage both the connection details and API
session token. This allowed the data to persist and be efficiently
checked before every API interaction. When the expiretime
was reached, a process to _reestablish the token was kicked
off.
def _reestablish(self):
"""
Use this to peroidically re-establish your
API Key. During long running API queries
you'll end up with an exipred token and receive
401 errors.
This check is triggered to handle those.
"""
if self.expiretime < round(time.time()):
l.notice(
f"API token session has expired. Expiration time: {datetime.fromtimestamp(self.expiretime).strftime('%c')}")
object.__setattr__(self, 'token', _token(test=self.test))
object.__setattr__(self, 'expiretime', round(time.time()) + self.duration)
object.__setattr__(self, 'headers', headers)I’d never worked with manipulating the loaded object
object in Python, but a dataclass can be manipulated with values from
any function that has access to the object. I don’t know how
frequent I will use this manipulation, but it was certainly
useful here!
When I’m integrating data from several sources, there’s often an ugly combination of logic that will become unavoidable at some point. It’s along the lines of:
if ( A in set1 and setB ) and not (if A.1 exists and ( A.1 == 2 )):
. . .Gross.
I had some really problematic code that was causing issues and went
looking for design patterns in Python that would help to manage just
this type of logic. I found a clever feature of Python: all().
Here’s a readable chunk of code from the linked site:
>>> bool_exps = [
... 5 > 2,
... 1 == 1,
... 42 < 50,
... ]
>>> all_true(bool_exps)
TrueMy real world implementation is a little more…involved. I’m sure
there’s performance advantages, but it’s also really nice to be able to
print a dictionary in debug logging that would just show
why the conditions aren’t met. I think I’ll use this for to
create better auditing/reporting during evaluation processes.
if item in list(self.data.SUBDICT.keys()):
state_check = {
"item_id_exists": True,
"item_unused": True
if self.data.SUBDICT[item] not in comparision_dict
else False,
}
else:
state_check = {"item_id_exists": False}
if all(list(state_check.values())):
l.info(f"Beginning delete process for {item}: {state_check}")
self._checkitem(item)mail can easily submit an email to a configure relay or
mail server and provide some basic features. Presentation however…that
requires some work. There are clear advantages to plain text, but
anymore I really prefer to have a nice looking list/report that I can
easily review. I don’t use emails for monitoring as much as I
used to – instead it’s typically reporting – which should look
nice!
In Bash you can submit mail direct to the sendmail
program. It takes a formatted email message, parses, and handles it.
(
echo -e From: serviceA@iamwpj.com
echo -e To: wes@iamwpj.com
echo -e Subject: Long running jobs
echo -e Content-Type: text/html
echo -e "
<html>
<body>
The following jobs took longer this week:
<pre>$(
echo "${results[@]}" | sed 's/ /\n/g' | column -N "JOB,ID,START DATE, EXPECTED RUNTIME, ACTUAL RUNTIME" -t -s','
)</pre>
[$(hostname -f)] - $(date)
</body>
</html>
"
) | /usr/sbin/sendmail -tWould produce a message like this:
The following jobs took longer this week:
JOB ID START_DATE EST_RUNTIME RUNTIME FeedMe 24 2023-09-08T11:35 12s 94s[feedme.iamwpj.com] - Fri Sep 8 17:51:14 CDT 2023
If I tried to send a plain formatted table in Bash, I can’t control the plain text table – the email viewer will simply output the text in the system font:
The following jobs took longer this week:
JOB ID START_DATE EST_RUNTIME RUNTIME
FeedMe 24 2023-09-08T11:35 12s 94s[feedme.iamwpj.com] - Thu Nov 2 11:32:37 CDT 2023
This pattern is easily reproducible and looks nice in scripts. It means I can simply drop my variables and subshells directly into the HTML layout. In the Bash world, I consider that a big win.
The one major detraction here: you cannot send attachments (well…not reasonably at least IMHO) using this method
I love shell parameter expansions. I think of it like the regex of Bash variables. It’s ugly, relatively unreadable, but still worth it.
Let’s look at my favorites:
If you suspect you might collect a null variable that
you don’t want, Bash can allow you set a default:
$ var=
$ : ${var:=DEFAULT}
$ echo $var
DEFAULTI use this a lot to manage subdirectories and dynamically specify
either a file or it’s containing directory without
having multiple variables or using awk to parse out
components.
$ data_location='data/folderA/mydata.txt'
# Directories:
$ echo -e ${data_location%/**}
data/folderA
# File
$ echo -e ${data_location##*/}
mydata.txtFrequently, you’ll get a resulting variable from an external source or something internally that needs to dynamically update. This means you’ll need to update the value within a variable. Rather than overwriting the variable, parameter expansion can replace components:
$ data_location='data/folderA/mydata.txt'
$ echo -e ${data_location/my/your}
data/folderA/yourdata.txtI’m sure there’s more, but these just hit me right today.