Latest picoevent news

12 hours of flow

john: June 16, 2019, 5:24 p.m.

Well after a trying week, I managed to make a really great addition the PicoEvent framework. Good place to store your precious API keys for Google, PayPal, AWS. Interestingly enough I got two auto vulnerability warnings from GitHub that applied mostly to ElGamal, I was never really a fan of the ElGamal Public Key infrastructure because RSA is rock solid and out of the patent dispute that brought about it's development. So never fear. Another interesting development was, I have been trying to keep the lines of code short so I can maintain test coverage. I found that a lot of my uncovered code in the database module were thing like hard coded SQL queries where the the exception was nearly impossible except or a database fault. (I had this function that locates the special superuser account a la Django.) You have to be really creative to throw an exception on a select for a single row that always exits. Now I am concentrating this weekend on converting the command line to curses uses Python3. My alternative text interface to Python's is not working as well as I hoped. Live and learn, but I'm thinking 50% coverage is pretty decent. Better to spend my time improving UI rather than trying to use some sort of crazy technique to get a database exception on a single row in a single table. Maybe block writes? I'm just going to figure out ncurses this Father's Day in the U.S.

Speaking of tooling

John Hill: May 29, 2019, 4:07 a.m.

One of the things that inspired me to make this into a separate open source project was that in the midst of my most hostile interview, ever. I refused to do the code test which was about four hours of rote Django services on Codality. And then the, uh, CTO proclaimed that I could not use a debugger. (I consider this like taking a driving test blindfolded) I quit after about 2 hours told the recruiter I was done. Moved on with the interview and when I was telling him about this project he said it was the only interesting thing I had to say. Sometimes the nastiest people give the most honest feedback.

Shilling for PyCharm Professional

John Hill: May 29, 2019, 3:36 a.m.

Now I was not the first person to adopt PyCharm, I was formerly a fan of ActiveState's Komodo IDE for a number of years, but since it doesn't integrate with Kite, I can't consider it here. I was intrigued by Microsoft's Intellisense AI which promised to bring greater productivity to beer making and combined with Kite, well worth exploring. My biggest issues are that for the PEP8 lint to work, you must first save the file. Also, there is easy refactoring tool tool to correct your PEP8 misdemeanors before you save your file. On the positive side it is free, combines Intellisense with Kite which is awesome. I also think it integrates with Git better. Definitely preferable to Atom, but given the dearth of Python developers, it doesn't quite compete with PyCharm yet.

Finally got my code up on GitHub

John Hill: May 15, 2019, 8:54 a.m.

It's been 11 days since I started it. I guess it's always hard to put something you made out there for ridicule, but this is helping me level up in Python 3.6, and almost everyone in the business around town in Central Texas is trying to build overly ambitious Django sites and integrate Amazon Kinesis for Analytics somehow. Django is pretty solid compared to things like Wordpress. It was quite a rush getting out into the world as something more than another Wordpress site. I have a couple Wordpress sites that are sort of stagnating. I can't even be bothered to check on them much anymore. They don't get hacked, but they get a lot of automated SPAM. I have a few interesting ideas to keep the spammers off of I like that it's a sort of a new type of forum I haven't seen before. Coming out much better than I expected, I could see it replacing Wordpress for me. It's funny, one of the reasons I got into microservices was that in 2015 I wanted to sell some software using Bitcoin to keep a few more bucks that I was aggressively marketing to make. I didn't see any reason to spend the sort of time it takes to develop a real website. And after my experience with, even with the incredible amount of tooling that has come along in the past few years... Web development is still tedious. Basically, if you want to check back at my GitHub in 2015 (which no perspective employer has ever bothered to do) the idea was to use Wordpress for the presentation, and provide advanced financial transaction support, and then integrate using REST and jQuery. Nobody ended up buying anything with Bitcoin, they prefer to pay with credit cards, but I learned about database replication. Wasn't much of a tester then, but I'm very careful. :)

Working on getting the Django site deployed to the Internet

John Hill: May 15, 2019, 8:04 a.m.

As hard as anything else I've ever worked on. Django should come with a warning label.

On a high level picoevent provides

  • Fast event logging/analysis (BI analysis functions can use the read-only replica, as not to affect write times.
  • Complete open source stack, initially designed to work with Flask/nginx/uWSGI with MySQL 5.7.
  • Simple lightweight code that requires few dependencies, which means:
    • Good unit test coverage
    • Portable across Unix systems (requires Python 3.6)
    • Inherently fast, stable, and secure
    • Extensible

picoevent is an open source business intelligence tool designed first and foremost to cover what I've found is a weakness in Django. Now I have had a huge affinity for Django since my first Django project which was a volunteer thing for a sailboat reservation system. It's pretty amazing that Instagram is also based on this framework. It's also sort of amazing that this is the sort of thing you're asked to do for free for (supposedly a fair shot) at getting a programming job, these days.

Django works great for providing fine grained access control to information, at an incredible rate of productivity for web development. One thing that has always amazed me about Django more than the fact that the likes of Instagram, use it is that many people who are not engineering managers are impressed by the speed in which things are accomplished.

I actually started this particular Django app as an exercise to prepare for a consulting job where I was told there would be Django. In fact they were using Tornado to build microservices, accomplishing almost nothing over months at a cost of millions of dollars. Engineering managers are easily impressed when you think about it.

One problem with event logging for business intelligence (what Amazon calls Kinesis) is that you have to carefully design the database because it often gets very large, very fast. Totally at odds with the Django approach of having SQL codegen. This is unfortunate, because with good business intelligence you can focus on the parts of the website that people are engaging with, and not waste ever precious development time on features nobody is engaging with.

Fine-grained Access Control + Business Intelligence = SaaS

The basic idea behind the picoevent system is that a non-programmer can define the events (a 'business analyst') which can be registered by a programmer not even particularly familiar with the codebase by inserting a single line of code which registers the event. picoevent is deliberately designed to be small and open ended (first of all to make it faster and inherently less buggy) The funny thing about write loads is that unlike read loads you can't really scale horizontally as straightforwardly because of synchronization overhead.

Inside the members area there is discussion of more advanced techniques of analysis as well as a marketplace for advanced extensions (unofficially supported) to picoevent. As well as discussions about how advanced Python with Frameworks can help the modern developer succeed in the ever more competitive world of web development. (by developer, I don't mean SWE II at AirBnB, I mean the developer that has to beg for their job every morning at a 'stand-up')

The official GitHub project is going to focus soley on providing an extension to Django, so please do not post issues regarding a feature that would not be relevant to someone trying to optimize the development of their Django site.