Python

Trying to Help Other Learn Programming

by

Having worked in the industry for a very long time, sometimes I forget how much I have learned (and forgotten) over the years. What seems trivial now, was once a complex idea that I had to wrap my head around, and after learning the fundamentals, there always seemed to be more to learn. This was at, in my mind, a golden age for learning computer systems and how to code. Installing Linux was not what it is today with flashing installers and disk configurators. Heck, X-windows did not start by default and you had to fiddle with configuration setting for ages to get a gui environment up to work in.

Now, I am not saying that we should return to the days of old, but it brings to mind that learning some of the core fundamentals is not something you get just by trying to get a system up. You have to go through different programs, and make the decision to move forward with learning the command line, and system fundamentals. And, with all the coding boot camps and quick starts, on occasion I am amazed anyone can find a good point to start.

That being said, I have been asked by some people that I know to help the gain a better understanding of the fundamentals of programming. There was the direct request to learn using python.

Python is a flexible language that you can do most anything in. And by anything, I mean almost anything. I started using it because it was a language that was available on Solaris that a co-worker did not want to use, and we at first did not want him on the project. My understanding, and knowledge of Python has changed a lot in the 8+ years since we decided to use Python for an internal company project.

That being said, it is a multi-disciplinary language. It is used for web development, system administration, machine learning, scientific studies…. In addition to this, other than the strange white spacing it uses to know what is going on, it provides a decent starting point to begin working with other languages. I have to admit, if you want to learn the true ins and outs of programming at a system level, Python will not get you there. For that you will need to dust off some books on C/C++. Although, I have heard that Rust is starting to replace some of that. Out of scope.

Back to Python and getting started.

I wanted to find a resource that would cover a broad range of topics when it came to Python, and also provided real world examples. When I was in college we spent a year going through a book on C++. That was difficult for the students, and many of them we lost along the way. And, after that year, I could not really write anything that would produce something that I could show to anybody. If it were not for the money I was forking out for college, I might have said what is the point.

After some quick searches, I chose a book to use. I decided to go with Python Crash Course, 2nd Edition: A Hands-On, Project-Based Introduction to Programming. (FYI) I don’t get any money if you use that link. The reason I chose this book because it has you actually do something that you can see results from.

Over the course of the book, you will work on 3 projects. I am sure they are not the most advanced projects, but they provide a foundation from which you can then launch your own projects from.

  • A video game utilizing Pygame
  • A Data analysis program with Visualization
  • A Web Application based off of Django

For a lot of developers, that is some great basics to cover. True, it does not cover writing a serverless application on AWS with Lambda and API Gateway, but, it goes about teaching a person how to think about an idea and implement it.

Note: This is all speculation. I am working through the book now with 2 people, but wanted to track my experience with it as I went through. But, I think one of the key factors is working through the entire book, and not skipping. The reason I say this is because I know a number of self taught developers that while excellent at writing code, do not know how to communicate their ideas with others.

Hopefully, by working through a book like this, it will teach enough of the fundamentals and language so that both of the people I am working with will be able to advance their careers.

Should I stop work on CfnMason or any project?

by

Off and on for years, and at various companies, I have developed various tools to manage complex AWS CloudFormation templates and stacks. This came out of the lack of tooling that was associated with CloudFormation itself. It was not that CFN was bad, (it is notoriously picky) it is just that it was designed with the intention of being a way to treat your infrastructure in code based manner. That is really not true. CFN was created as a templating language with defined spec.

Because of these limitations, I have built closed and open source solutions to manage the complexities that involve working with CloudFormation. Recently I even started revamping a tool that I wrote years ago to manage complex CloudFormation Stacks. This update was done on the behest of a few people that actually utilize the tool and wanted to be taken from the messy state it is currently in, into something that could be tied into their current applications. It was from this that I began working on CfnMason as a python module.

However, recently I started working with CDK. CDK is Am,azon’s CloudFormation Development Kit. My first thought was that it was going to be horrible, and why would anyone ever use it. Now, this was before it was a fully supported implementation and was only really viable when used with JavaScript. And, don’t get me wrong, I don’t hate JS, but I do most of my coding in Python these days. So, when I finally had a chance to use it for work, I found that I really like it, and that it was actually an excellent tool.

So, that brings me back to the original question. How do I know when it is time to stop working on a project? The main answer in my mind is you have to figure out that for yourself. When I started writing this, I was pretty sure that I was going to say that I am no longer going to be working on updating CfnMason. But, as I wrote this, I realized that not everyone is going to be able to move over to CDK. There are probably thousands of CFN stacks that have been created over the years, that require updates and tweaks, that are not a good fit to move over to CDK. As of yet, I don’t know of a way to take a template that is in AWS and convert it into a working CDK script so that you can develop on it from there.

This is why writing is sometimes the best way to find an answer to a problem, even technical ones. At face value, there are a number of projects that seem like they should just be discarded and never used again. But, once you analyze the situation properly, you might realize that there is a reason to move forward with development of a seemingly dead solution. It might even be to use it as a growth platform. Or, it could be that while there are new tools available, that for some, older and simpler tools are also still needed.

So, at the end of the day, what started as a note to say that I am no longer going to be working on CfnMason, has been turned around to me stating that I am going to try and get it done. Ha, yeah, even I laughed at that. Although now that the nation is in lock down, there is more of a chance that I might get it finished.

How to configure environment for Python Poetry based project.

by

How do I get started?

I recently switched over to Poetry as a package manager for my project CfnMason. ( CfnMason is a tool related to CloudFormation stack management, but you can read about that on the Readme as it is updated. ) The question is, how do you use Poetry when you are working on a project across multiple machines and operating systems. I guess I am going to attempt to address the issue.

So, you want to join a project, or work on a project that is using the Poetry dependency management tool. Great! But, how do you get the requirements setup for the project so that you can start working on it? How do you know which version of Python to use, which packages to install, how to build the project, or how to run the test suite?

This is an issue that I was facing, but it was not with another project, but my own as I was switching between machines. Now, I do know some of the answers to the questions above, but I was still stuck as to how to setup a project on another machine. As such, I decided to walk through the process of coming onto a new project and determining how to work with it. The project that will be used for this walk-through will be CfMason, at tool that manages some aspects of building and deploying CloudFormation stacks on AWS.

Hopefully the project that you are working on has a Readme file. Though, to be fair, documentation is hard, and is often the last thing that is added to a project. If it does, you should be able them, but if they are not provided, then the following steps are the way that I would go about working on a project that uses Poetry for dependency management. Oh, and as a note. I am making the assumption that you already have Poetry installed.

Steps to work on Poetry based Project

  1. Determine that the project is using Poetry
  2. Check the version of Python that is needed
    1. Validate local Python version
    2. Install if missing
  3. Create a virtual environment for building the application
    1. venv — for packaging and validation
    2. venvdev — for building the package
  4. Install project dependencies
  5. Build the project
  6. Run tests

Determine the project is using Poetry.

If the ReadMe does not tell you that the project is using poetry, then there is a quick way to find out.

  • Look for the file pyproject.toml in the base of the project.
  • Open the file and look for the following line
    • [tool.poetry]

Provided you are able to find this file, and line, then the project is using Poetry.

Determine the version of Python

One nice thing about Poetry is that it has a defined location to identify the version of Python. I am a big fan of this, as the difference between different versions can cause major problems. Take for example that reserved keywords changed a bunch between 3.6 and 3.7. The steps to follow are as follows.

  • Open the toml project file pyproject.toml
  • Find the supported python versions in the [tool.poetry.dependencies]
  • Find the line starting with python, and find the supported versions.
    • Ex. python = "^3.6"
  • If you don’t have that version of Python on your system, install it.

Setting up a local Virtual Env

I am a huge proponent of using a virtual environment for each application that I am working on. In some cases, I will have 2, one for including all the development modules, and one for just the ones needed for the application to run. Since Python 2 is pretty much EOL, I am not going to spend any time on how to setup a virtual environment for Python2. Instead, this is all dedicated to Python3. And I can only guarantee this on Python 3.6 or later.

foo$ python -m venv venv-dev
foo$ python -m venv venv
foo$ ls -ld venv*
drwxr-xr-x 1 foobar 197610 0 Nov 24 17:02 venv/
drwxr-xr-x 1 foobar 197610 0 Dec 25 14:19 venv-dev/

Install project dependencies

For this last part, you need to activate either of the Python Virtual Environments and then run the install code from there. This is only if you really want to install it both ways. If not, then you can just create a single virtual env directory and just install all the dependencies.

Install all the dependencies, even the ones needed for development.

foo$ poetry install
Installing dependencies from lock file


Package operations: 10 installs, 0 updates, 0 removals

  - Installing more-itertools (7.2.0)
  - Installing zipp (0.6.0)
.....
  - Installing cfnmason (0.1.0)

The other option is to just install the libraries needed to execute and run the module. I would almost prefer if it defaulted to the method below, but it works. 

foo$ poetry install --no-dev
Installing dependencies from lock file

Nothing to install or update

  - Installing cfnmason (0.1.0)

Start working on the Project

That is it. You should be up and running. At least to the point where you can get started with the project. Moving forward from this point will rely a lot upon how the project is setup, and how well it is documented. But, the big factor is that you can now start working on it while using Poetry, or you have the foundation to work on a project across multiple machines.

Setting up the cfnmason project for Python

by

Cfnmason is yet another tool that can be used to manipulate cloudformation stacks. It is not designed to be a replacement for CloudFormation like Terraform, but as a means of making building and managing them easier. I had written a version of this ages ago in Ruby, but with most of my work now being in Python, I am creating a new version in Python. As I have never created an exportable Python package, this will track the process of building and releasing a new PyPi package.

Oh, and to keep things interesting, I am doing this on a mix of Windows 10 and Linux.

Creating the default layout using Poetry

The first thing that we need to do is to create the base package. I could do it by hand, but I want to try out the Poetry package and see if I will hate the decision later. 

c:\dev> poetry new cfnmason
...
c:\dev\cfnmason> tree \f \a

|   pyproject.toml
|   README.rst
|
+---cfnmason
|       __init__.py
|
\---tests
        test_cfnmason.py
        __init__.py

Init a new Git repo and add a .gitignore file

These are more civilized times. As such, I almost always create a git repository when I am working on a project, even small ones. They may or may not be public, and it can fluctuate on which platform I use to host my code. This time I am opting for Github, and have uploaded the code.

I have been using gitignore.io to generate base .gitignore files for ages. Type in a few operating systems, the language you are coding for, any IDEs, etc, and you can have a useful .gitignore file right out of the gate. Sure you can do it by hand, but this is quick and easy, and it can always be edited later.

Modifying the Readme file

Poetry starts you with a README.rst file. I don’t know about you, but I have been working with MarkDown for ages. It is common on a number of platforms and there is support for it in a number of editors. I understand the RST files are really designed for technical documentation, but I can burn that bridge later. For now, I need a decent starting point.

We could have started by writing out the template by hand. This would have been long and tedious. Instead, I am using a template. By using a template, I am up and running quickly. I can add and remove parts that I do or do not need, and hopefully, I will not forget a major part. As this is the first pass, I am not going to update the entire thing, but at least get the project name in and the fact that I am working on it.

The license file

The last thing needed before I start working on the code is the license file. Which license you choose is up to you. Personally, I like to use the MIT license. It lets people do what they want. As this is an opensource project, I feel that people should be able to use it as much or as little as they want.

The easiest way to do this is using the Github console. Just add a new file via the web interface and name it LICENSE or LICENSE.md in all caps. Then you will have the option to choose from a list of known licenses.

That is it and next steps

Hmmm. This took longer to write than the work to get the base package up and running. And, that is going to be expected. But, it should also show some of the thoughts and considerations that are needed when creating a new project. Especially if it is going to open for the world.

With that I will leave you to it. In the next couple of days, I am going to start adding in the libraries that are needed to begin working on the project. As I have written this in Ruby before, I have a rough layout in my head of how I want the application to work. And, I know what libraries are needed to meet the core functionality.

The question that I have for myself is if I should take the time to ensure that I update the design docs. Having a design doc is a great way to ensure that you are not missing any features. I am going to have to sleep on it.

How do I setup a new python project?

by

Python prides itself on there only being one best way of doing things. However, if you have ever had the desire to create a package that can be distributed in python, then you may have run into some frustration. Out of the box, there does not appear to be a single clear concise way of creating a new project. After hearing over and over again, that the correct way in Python should be obvious, in this area, it seems that this is definitely not the case.

There should be one-- and preferably only one --obvious way to do it.

— Zen of Python

However, it does seem that there are a couple of solutions that have been put forward to solve this problem, and none of them have officially been endorsed by the people that manage Python. You can do it all by hand, and I give you credit if you do, but this is more work than I want to keep up with. I started to walk through the documentation to use setuptools to configure a package, and almost wanted to cry. Not really, but when you cannot give a clean concise way to build and distribute a python package, there is a problem.

The only thing that is clear is that you should have a setup.py file. Other than that, you are on your own. As I said before, the documentation from setuptools is laughable. Maybe if I did not mind reading pages upon pages to get a simple package started I would be OK, but I don’t have the patience for it. I thought that it would make sense to look at existing packages, but once again, everybody seems to setup their projects differently. Don’t get me wrong, there is nothing wrong with developers taking their own routes to setup a package, especially after seeing how little guidance there is. Maybe packaging was an afterthought for Python.

Having used Ruby in the past, I was sure that there had to be packages out there to assist in setting up the initial package structure and environment. Now, I do not expect it to write the code for me, but the basics of what is in the package, file layout, author information, etc. And after a bit of searching and head scratching I finally found what I was looking for. Well, at least I got a bit further down the rabbit hole. I had honestly thought that by this point I would be working on migrating my app from Ruby to Python, not trying to figure out how to create a package.

What tools are there?

It seems that upon first glance, the top three tools for creating Python packages are Poetry, Pipenv, and Hatch. And when I say creating, I mean creation and management of the packages. There is always doing it by hand, and maybe I will end up there, but that goes against the grain of automation.

I have spent a bit of time looking at these three options, plus managing it by hand. After looking at it for a bit, I think that I am going to throw Pipenv out the window. The lack of proper documentation is a stopping point for me. Also, it seems like there is a fork that is now responsible for the actual development, and not the original source itself.

That leaves me with Hatch and Poetry. Decisions, decisions, decisions. I am not sure which route I am going to go down.

Which one do you choose?

I think that I am going to start by creating my project using both methods. Early on, it should be simple enough to create the source and copy it between projects. The real question will be, which one will make management easier in the long run. That means, in my next post, I am going to start building, or more accurately, rebuilding cfmason.

Within a week or two I should be able to make my decision. But, I am going to build them out from scratch both ways, and record the process. Heck, that will probably be harder than the coding itself. Documenting this stuff is not easy.

See you in a week.