RewindNs

 While taking online classes via Zoom, sometimes the slides go really fast and I cannot finish writing. In such times I wish I had a magic button to go back a few seconds and hold it right there until I complete noting down the slide. I know there's always the option to record it all (OBS works fine) and go to whatever time in the video and take notes from there but having something akin to Nvidia instant replay would be neat. That's the concept behind this program.

I chose Visual Basic 6 (or the way I call it - vb6) to code it up because it had been some time that I had not used it. It used to be my go-to language/IDE for making anything and everything before 5 years or so when I made the switch to C# and the rest. I remember the IDE being super snappy, the compiled file size (12-20 KB for a normal windows form/standard EXE application) super small by today's oversized standards. It was such a breeze to get things done without worrying about classes, objects and all that convenience. And doing things you were not supposed to be doing in vb6 (multithreading, executing shellcode/assembly, calling undocumented win32 API, etc.) and getting them to work (after shooting yourself in the foot a hundred times of course) was such a treat. I guess I wanted to feel that again. I don't think there's yet been another language that has been able to occupy that sweet niche for me. PureBasic and Delphi/FreePascal Lazarus come somewhat close but they're a significant compromise.

Anyway, I started it yesterday (~6PM perhaps) and finished it today afternoon (~2PM). It turned out to be a bit more work than I had thought it would be - I guess my observation that one has to underestimate the effort it takes to complete a project to actually embark upon it is something that I will never learn from. It's done now and here I am writing this up at 8:10 PM. There's still a few nice-to-have features I'd like to add but this version works just fine for what this program is supposed to do.

Flowchart of the program

If you can't make the flowchart, in a nutshell, the program maintains an array of screen's memory DC (Device Context) - basically a screen capture - and it paints the first element of the array over and over in a loop over the screen when the user presses the 'J' key twice, until the user presses the 'L' key twice. The array's length is equal to the history timespan divided by the sampling interval; so if the history timespan is 5 seconds (meaning save a history of 5 seconds i.e. 5000 milliseconds) and the sampling interval is 100 milliseconds (meaning the screen capture is performed every 100 milliseconds), the array will have a size of 5000/100 = 50. This array is maintained so that only the most recent 50 captures are present at any given moment. This means the array is first filled normally up to the end, then the elements are pushed one position to the left each time a new capture is performed so that it can be placed at the last position. The captures in the array move from the oldest to the most recent left to right (start to end). Here's how it looks. You can right click on it to hide the window.

Here's the GitHub link for the project. Here's the download link.

Bulk emails with SMTP

I was exploring ways to send emails in bulk without having to pay for a service. I did some experimentation using Python's 'smtplib' library. Here, I present some of my observations:

1. I tried sending an email with a Discord server invite to a little over 150 people (retrieved from the 'To:' field of an email in my Gmail inbox) by simply pasting the email addresses in the 'To:' field of Gmail's web interface, but it didn't reach them. It wasn't going to their Spam either. Gmail didn't give any error either. So I thought Google must have delayed the request owing to the large number of recipients. Later that day(~6 hours later), I found that the email had popped up in my Spam(my second email address was among the recipients). This is why I thought sending individual emails to the list of recipients via SMTP would be my best bet.

2. I didn't want a commercial software and I didn't find any widely available safe, free option. I wasn't planning on making an end-user focused software either. So, I chose Python with the smtplib library. I made the following observations from my experimentation. Note that I made a new email address corresponding to each of the following SMTP servers I tried.

a) Yahoo's SMTP server blocks email send requests upon sending 10-15 emails back to back (throws exceptions - SMTPDataError, ConnectionRefused exception, etc.)

b) Gmail's SMTP server easily sends a hundred emails (could be more, didn't test) flawlessly on the client side without throwing any exception. but it is not certain those emails will reach the destination, and when they do, they can end up in spam

c) Outlook's SMTP server easily sends over a 150 emails (could be more, haven't tested) but its behavior is complicated. If you haven't logged into your Outlook email inbox recently, and you try sending emails in bulk, it throws a very descriptive exception: SuspendedException or BlockedException or something along those lines, meaning your account gets flagged and temporarily blocked. To remedy this, you just open your Outlook inbox in a web browser, and let it verify your authenticity - usually asks for a mobile number, just fill up the access code sent via SMS and it gets unblocked again. Then, it's pretty well behaved after this. I kept batch sizes of 50 emails back to back. Then I sent a single legit email using the web interface of Outlook and went back to the script again. Worked like a charm. But, although the email was being reported to be successfully sent on the client side(using the SMTP script), the email could end up as Spam on the recipient side. To remedy this, I adopted the approach explained in (d).

d) While sending emails via SMTP(not the web interface), including an external link (in my case, a Discord invite) in the email body was enough for the receiving Gmail client to mark the email as Spam. However, when I removed just the link with the rest of the message intact, it appeared in the recipient's inbox right away. Then I tried sending the same email with the link included again and it was no longer marked as Spam, it went straight to the proper inbox! So, I figured it's better to first send a warm-up email not containing any link and then to immediately follow it up with another email which does contain the external link. But sending these two emails back to back was still getting marked as Spam. It was probably due to the short duration between the two emails. So, I switched to first sending a batch of warm-up emails to a batch of recipients, waiting a couple minutes then sending the actual email to them. this worked very well with Outlook's SMTP server. Haven't tested with Gmail's.

Minor observations:

i) Yahoo requires generating an 'app password' for use in SMTP mailing; the default login password doesn't work.

ii) Gmail doesn't require generating an 'app password' but does require enabling 'less secure apps' from Gmail account settings. The same login password is used.

iii) Hotmail doesn't require 'app password' or anything, just your regular login password will do. It just emails you to confirm if the 'suspicious app' that tried accessing your email account was you. You can confirm it through a web interface and you're good.

iv) I had tried Yandex.ru as well, seeing as it didn't require mobile number to sign up but its SMTP server didn't behave too well. Btw it also requires you to generate an 'app password' for using SMTP.

v) Protonmail doesn't have a public SMTP server, so it's not as simple as the other email providers to send emails programmatically with it.

vi) The SMTP message headers are difficult to get right, for example the "Subject" header is sometimes recognized by a receiving client while some others don't detect it and display (no subject) instead. It also changes with the SMTP server you're using to send the email. The SMTP header structure seems to be interpreted differently according to the SMTP server you're using.

Here's the corresponding script.

Update/side note:

The conversion rate seems to be hovering at around 10% in this campaign.

NepseWatcher

 I've been investing in stocks recently. It has been an alien subject to me for most of my life. I only got into it owing to a few chats with a childhood friend of mine. I had zero understanding of the stock market before that. But about a month back, that changed. Because of the supposedly free time that I have right now since quitting my job just over a month back and due to the lockdown imposed against the second wave of Covid19, the topic of shares and stocks had my mostly undivided attention (putting aside the ticking time bomb that is my Masters thesis). So, in researching this vast subject, I thought it could prove useful to have a program that could scan the listed companies for possible trading opportunities. I just could not find anything for NEPSE for this purpose. 

Some of the relevant websites I visited did have various screener parameters but not much towards my expectation. So, I embarked upon a journey to create a program that would analyse stocks the way I wanted. Starting around the beginning of this month (May, 2021), I began in earnest. As I hammered out one feature, I went on to add another - all along also continuing research regarding technical analysis. YouTube was a great resource as always. I watched a bunch of technical analysis and swing trading strategies among which an hour long video by Bhanushali is something I have to mention here owing to how much of an influence it had on my take on stock trading. Also, the reddit sub r/nepalstock was a huge resource and a guiding hand to me as a newbie in the field. I also discovered a lecture series by Rachana Ranade on technical analysis which I found was really good. (I'm yet to finish the series.) 

The first feature I wrote was to sort the companies with respect to their Moving Average slopes of closing prices. I then added Candlestick pattern recognition with six different patterns. The most recent one I did was a Return On Investment (ROI) analyser. I also thought a watchlist would be a great addition to the program since I find myself going to Meroshare and my portfolio spreadsheet constantly to check how the companies are doing. So, a couple of days ago I also added a watchlist capability that calculates the net profit, average ROI, etc. all incorporating the hidden costs such as the brokerage commmision, taxes, etc (thanks to sharesansar's share calculator).

Perhaps this could have been a monetisable product were it not for the fact that I do not have access to the prohibitively expensive NEPSE API. All data used by the program are either extracted via web-scraping or reverse engineering third party websites' APIs. As a result, things can break at any given moment. Nevertheless, the program works just fine as of this writing. [Except for the candlestick chart URL used in the browser window. I previously used nepsealpha.com/trading/chart which started blocking any Selenium-driven browser soon enough. I've since switched to systemxlite.com's chart which uses a token-based approach to enforce a time-limit for session validity. So, you have to login to your account on systemxlite.com and go to the technical analysis chart for a functional URL which then you can paste on to the URL bar of the Selenium-driven browser window spawned by the program]

The details of this 3 weeks+ long adventure have been documented in my MDD.txt file @ GitHub

Download here

Screenshots:

PhoneMapper

I recently came across a neat Android app called MacroDroid. It is an automation utility that lets you make macros - basically scripts, that get triggered at different events; very akin to AutoHotkey in Windows. There were a lot of features in this little app. I was pretty impressed. I tinkered around with it for some time and one day I thought why not make a location tracker for myself. So, I made a macro that would log the location, along with various other properties of the phone (the battery level, uptime, etc) every hour to a local file on the phone. Soon, after a couple of days I had accrued maybe a hundred records. Then I wondered how it would look if I was able to visualize all that data. I searched around the internet for anything of the sort but any program able to do that would have to be fed the collected records in a set format - one that it expected and that would entail some degree of data-cleaning. So, I decided to make an app of my own for the express purpose of reading the log recorded by my macro and displaying it on a map.

For the aforementioned purpose, I instinctively went to the Google Maps API but turns out it cannot be used without a billing address. Even people who do pay to use it have qualms about how expensive it is compared to other, similar services. So, I investigated some more and decided that OpenStreetMap would be my best bet. I made a boilerplate webpage using its tile server in leaflet.js and was starting to get the hang of it - starting from drawing a map centered at a given latlong location to placing markers on the map but I had to be able to read the log file uploaded to Google Drive from my phone before drawing the map. At first, I thought maybe I had to use the Google Drive API for it as well but I found this post. So, the direct download link problem was set. But when I actually proceeded with the fetch() API in JavaScript, CORS held me back. Apparently, the browser doesn't let reception of a resource sent by the server if the sender is not the same origin as the server. In my case, I had launched the web page locally, i.e. by opening the corresponding html file, so Chrome would not let me view the log file from Google Drive because the request for the file originated from a file:// source and not an https:// one and the CORS policy that web browsers follow forbids that (if I'm correct). So, it was clear that I had to either host a website running this frontend code or alternatively opt for the Electron framework for a desktop app. Surely, that had to work, I imagined. But the last time I made something in Electron, it was absolutely massive. So, I steered clear of it. Instead, I chose C# - surely a language this popular and in such rapid development must have a very stable library for displaying maps.

Well, C#, I found out, while for sure having a lot of libraries for a variety of different purposes, was pretty lean in the area of drawing maps. Fortunately, I stumbled upon GMap.net, a mapping library available also via NuGet package installer that provides a convenient component for drawing maps in Windows Forms, WPF applications etc. Though the documentation was virtually non-existent, this was all I needed to get started. Everything went ahead smoothly in C# and now, a couple of days later, here I am. A functional version 1.0 is here and this is what it looks like:

The program should be pretty self-explanatory but I've also included instructions in the Help menu regarding the setting up of the log file URL and the format it expects the file to be in, along with part of the MacroDroid macro that I personally use.

GitHub

Download

My Musings During Development:

References:

https://stackoverflow.com/questions/44888219/windows-form-display-dynamic-google-map-instead-of-static-map-in-net

http://www.independent-software.com/gmap-net-beginners-tutorial-maps-markers-polygons-routes-updated-for-vs2015-and-gmap1-7.html

https://stackoverflow.com/questions/44992766/is-there-any-good-gmap-net-documentation-out-there-for-c-sharp-windows-form

http://www.independent-software.com/gmap-net-beginners-tutorial-adding-clickable-markers-to-your-map-updates-for-vs2015-and-gmap-1-7.html

https://www.c-sharpcorner.com/article/serialization-and-deserialization-in-c-sharp/

https://docs.microsoft.com/en-us/dotnet/api/system.io.filestream?view=netframework-4.7.2

https://stackoverflow.com/questions/3029675/listview-with-copy-paste

Feb 8, 2021 | 10:40 PM

-----------------------

I found that loading the table data from the contents of the downloaded file to the listview table itself wasn't that heavy computationally on the GUI of the app since I was using async/await for the download process. But still the app felt laggy during the population of the listview table. So I switched on the CPU profiler tool in Visual Studio Diagnostic Tools and it showed me that the majority of the CPU hogging was due to a seemingly innocent method for resizing the columns of the listview table:

listView1.AutoResizeColumns(ColumnHeaderAutoResizeStyle.HeaderSize);

All it does is it resizes the widths of the table's columns based on the header size/contents of the columns'. Amazingly, upon commenting out this command, the app felt way snappier. There was no visible lag or unresponsiveness while populating the table with the downloaded rows.

Feb 9, 2021 | 09:08 PM

-----------------------

Enabling AutoCompleteMode/AutoCompleteCustom/AutoCompleteSource properties of a toolstrip textbox seems to mess with its KeyPress event with the Enter key. Other keys trigger the event correctly though.

Feb 10, 2021 | 10:19 PM

-----------------------

Icon attribution:

<div>Icons made by <a href="https://www.freepik.com" title="Freepik">Freepik</a> from <a href="https://www.flaticon.com/" title="Flaticon">www.flaticon.com</a></div>

Feb 11, 2021 | 12:45 PM

-----------------------

I had originally envisioned a web app for this project. But because of the same origin CORS policy enforced by browsers, I was unable to download the log file from google drive while running the html file from hard disk. I then thought of Electron as an alternative but the last time I made something using Electron, the output executable size was massive(~70MB I think). So, I settled on a desktop app in C#. But in the end, its dependencies made it so that the final size of the output (the main program executable along with the dependency dll files and corresponding resources) is around 16MB. I now think that maybe the Electron idea wasn't that bad after all. I know it is a significant difference between 70MB and 16MB but the dependencies are a headache to distribute without an installer. And I don't have much experience of producing installers. I'd much rather have a standalone executable.

Feb 11, 2021 | 02:52 PM

-----------------------

I only learnt of GitHub Releases a couple of minutes ago. Gonna use it more often. Neater way of uploading your compiled binaries.