Follow-up to T-SQL Tuesday #113 – What Do YOU Use Databases For?

Wow, we had a variety of responses to the April 2019 topic of “What Do YOU Use Databases For?

I think the overall response to the question and the theme is both mixed and varied.

I have been struggling with the personal use of databases for a long time. Things I wish would have been easier but seems to just get more complicated over time. Ever heard of GDPR? Although we think we have absolute control and access to data about ourselves, we really do not. The right to be forgotten is NOT the same as having access to all of the data about ourselves in all of the systems before they disappear. Sometimes companies will delete your data about you before you ask.

Case in point: Microsoft HealthVault. It is going to die. And your data? Deleted. https://en.wikipedia.org/wiki/Microsoft_HealthVault

Tell me- if your personal physician or other health care practitioner wanted to access your HealthVault records 20 years into the future, how would they do it? After the service has long been terminated and your digital data destroyed? They cannot. Did you pull out your data before the service was terminated and put it someplace else?

Do you know who is responsible for digital data about you? It is the same answer as to who is responsible for your own data career.

YOU.

We are like a Stranger Living in a Strange Land (Yes, I am tainted by Heavy Metal. It is a part of me. And no, I am NOT ashamed. https://www.youtube.com/watch?v=UJsl-bB7lmk).

Digital records are easily created, changed, and deleted. We are living in a so-called enlightened and evolved age compared to the luddites who embraced physical paper records of data. A short time ago, I posted via Twitter my concerns about Wills and Testaments in this digital age and how often do people perform DR tests to access to our digital records, especially after we die…

We believe we have access and control to all of the data collected about ourselves but we do not.

In an intentional manner, this month’s T-SQL Tuesday topic was just that- when we choose to create databases for ourselves, but what do we do or use it for and why? How do we back it up?

The crazy part is that for the ubiquitous SQL Server .bak file we now have to worry about having access to the versions of the software to restore it. Today I can open up .pdf files from over twenty years ago. Can one do that with .bak files?

No.

I have to restore the .bak to an old version of SQL Server, then back it up again using a relatively newer but compatible version, then restore it again using a newer version of the software, rinse and repeat until I can see the data in the latest version of SQL Server. Cumbersome to say the least.

Believe me when I say there are non-technical people who don’t want to have to fiddle around with ANY of this stuff- they just want it to work and have some kind of guarantee that some time in the distant future their digital files will be able to be successfully opened and used by people they want to be able to open and use the files.

We use databases all of the time and dare I say it? (Dare, Dare!!!) Databases use US. Ever wonder how when you click on something at work and then you start seeing ads for things on your personal devices? Hm…

For example, why would you build a database schema and then both build and populate data via a UI when one could simple use an existing service to track your finances or fitness data? Several of the responses to this month’s post seem to echo that sentiment- why build when you can buy? However some folks do it anyway (like me) as a learning experience much like I did with our VHS collection. People are willing to pay good money to make pain go away quickly and efficiently. They don’t want things to drag out over time. Instant gratification to a problem via an app or existing solution trumps wasting time (!) building something from scratch. But alas, like the demise of Microsoft HealthVault, is one willing to sacrifice inputting today’s data into a system that could both die and not be able to extract one’s data out of it sometime in the future?

I keep going back to both the book and the project MyLifeBits. It rings a deep chord with me. I imagine it will continue to do so. We should be able to have access to ALL of our own personal digital data and decide what happens to it over time. When we control the data ourselves via our own schemas and systems, that becomes fairly straight-forward and yet is time-consuming. But when we outsource that data and storage we relinquish control and trust and hope it will be available to us or loved ones sometime in the future.

Without further ado, here are my observations and comments on this month’s T-SQL Tuesday topic:

Jeff Mlakar (@jmlakar | b)

Jeff talks about the skills we need to stay sharp by being able to easily spin up a virtual machine and jump right in. (On a side note, I got an e-mail just yesterday thanking me for creating a document to build a virtual lab for the 70-462 (SQL2012) exam. Time flies). There is a lot of value in understanding the entire stack in a personal application. It is just enough complexity to wrap your brain around but not too complex that one feels overwhelmed. Building and re-building hones our skills and provides insights into troubleshooting. Keeping an eye on income and expenses via spreadsheets is smart but ideally still need to maintain security and keep it backed up somewhere out of the home. He ends by learning to play with data by importing and manipulating large datasets. Thanks Jeff!

Kenneth Fisher (@sqlstudent144  | b)

Kenneth reveals he likes designing and building crossword puzzles using SQL Server. Awesome! More than one person on this topic has referred to ERD(s). (BTW, whenever I am at a client and before I start poking around with DMV(s), I like to restore a backup of a database and create an ERD. It provides a way to visually see tables and relationships. It also can show lack of relationships. NOTE!!! This can take a while using Database Diagrams in SSMS. One observation over the years- the tables with the most number of constraint connections to them are some of the most used tables in queries. Which tables, indexes and queries to tune first? The ones which are used the most.) The entire Head First series from O’Reilly has crossword puzzles at the end of every chapter. It is part of their metacognitive strategy to writing these books. It is a great way to re-enforce concepts and gets you thinking about things in a fun way. Now I’m thinking about how to build a 3D crossword puzzle in virtual reality so one could walk/fly around it. I like word art and shadows. Thanks Kenneth!

Kevin Chant (@kevchant  | b)

Kevin discusses the future of smartphone apps hitting databases in the cloud. Anybody with a smartphone is constantly generating CRUD queries. Ever think about how many queries are run by you in a 24 hour period via any and all apps? And the growth of data stored in the cloud is going to continue to grow. Thanks Kevin!

Tracy Boggiano (@TracyBoggiano | b)

Tracy shares she has a sports card collection database with over 50,000 cards! That’s cool. I’m curious how the pricing gets updated and captured- can you see the delta(s) of price changes over time? And is it a manual process or automatic? Interesting. It is encouraging to see people getting into databases at a very young age and being able to create something useful for themselves! She also captures fitness data. I’ve seen many people wearing fitness trackers over the years. Most smartphones also have some kind of default fitness app which uses the phones accelerometer too. Being able to quantify your health helps to keep one mindful about your current and future health goals. As the saying goes, if you can quantify it, you can measure it. Thanks Tracy!

Reitse Eskens (@2meterDBA | b)

Reitse talks about volunteering in speedskating. And the challenges of having to sync two applications for the data. Getting Windows Updates during a race, ARGH. (In the past I used to help out with Cub Scouts and AWANA Pinewood Derby races (sensor on release and on finish line) and different kinds of races. Generally each car will race once down each lane. So the software figures out how to organize all the racers. It had a Microsoft Access database back-end on a laptop). People are storing more and more personal data in the cloud but I think each individual needs to decide what works best for them. I am uploading a ton of .zip files of photos to my OneDrive and it….is…slow… And I like your tractor analogy too. Thanks Reitse!

Jason Brimhall (@sqlrnnr  | b)

Jason echoes others in use of personal databases to trace fitness and finances. I really like the idea of dreaming up small things to try and test and do all over again. His link to Steve’s post is spot on- I would add this applies to speakers in the data community as well. Do you know how hard it is to convince people to present? Our community is filled with a lot of really smart and experienced people but a lot of them have no interest in public speaking. They are either not comfortable with it or don’t see any benefit. To be blunt: if we are not learning something new, we are slowly dying in our craft. He ends with a call to use automation (including inside a database) to sustain your work and I would add your sanity. I’m still slugging through a Python book called Automate the Boring Stuff With Python as a way to a) force myself to learn a new language and b) learn something besides T-SQL to do useful work. Thanks Jason!

Eugene M (@SQLGene | b)

Eugene has played A LOT of games. Games always have been a social experience played with others. He does show something that is prevalent in ALL data-entry systems- quality and time to perform data-entry. I can’t tell you how many times I’ve see free-form fields used in applications and then people wonder where did we get all of this dirty data from? Can we clean it up without breaking things??? I like how they play video games together too and that is important. And if kids are in anybody’s future, I would advise ALL of you to play both board and video games together. It creates some of the best memories one will have for life. Thanks Eugene!

Andy Levy (@ALevyInROC | b)

Andy is into geocaching using GPS devices wherever he goes. (NOTE!!! My twenty year old Magellan GPS stopped working on April 6th. It is now a museum piece). If I had to find a fake pinecone and money was no object, I might spring for a FLIR for my phone. Hiding in plain sight, I love it. You should see the number of caches he has found and some of the metrics, wow! I want to get back into it but need to be aware of rattlesnakes where I live. More ERD diagrams like some of the other posts but this time from reverse engineering a SQLite database and then doing some PowerShell ETL into SQL Server. This is great practice for those considering getting into IoT devices too. Great stuff and a great excuse to get outside and out from behind our computers. Thanks Andy!

Nate Johnson (@natethedba | b)

Nate makes the case that he’ll get his hands dirty when he needs to but for simple things like an inventory list, just use a simple tool to get the job done in a short amount of time like Google Sheets. There seems to be an ever-present pressure to try to be on the latest versions or playing around with the latest technology for SQL Server. Personally I still owe a review for a Linux on SQL Server book that I’ll get around to probably by summer. I work in a Windows shop and all my home systems are Windows. But I do want to learn about Linux and containers especially after I’ve seen Bob Ward present on them. Magic! Just not a super priority for me right now. Physical to virtual machines is the first step before learning how to migrate to the cloud. Thanks Nate!

Lisa Griffin Bohm (@LisaGB_sql | b)

Lisa started using databases to help plan her first wedding! It is a lot stuff to track and you don’t want to risk hurting people’s feelings by forgetting something. Saving money by doing it yourself when it makes sense is smart. At least for address labels for Christmas cards, I have used Word’s mail merge program and Avery labels. More and more people just want to keep things simple like using Trello. Thanks Lisa!

Michelle Haarhues (@mhaarhues | b)

Michelle is an avid runner and loves to go on destination runs! While some apps do a great job of tracking runs, they don’t have an easy way to add additional things like destinations to see or store advice about items like health stats or run suggestions by friends. Moving from a spreadsheet to a database is not always easy. A training partner database has a nice ring to it. Being able to combine vacationing and running sounds like a great way to stay in shape and keep learning databases. Thanks Michelle!

BonzaOwl (@BonzaOwl | b)

Codename “BonzaOwl” needed to track vegan-friendly items electronically like in a database. When one is passionate about the data it takes on a certain level of importance. Being able to track food choices is critical for several people I know; there were some kids in my son’s Boy Scout Troop years ago that had peanut allergies. No joke- he could have an allergic reaction to something that got cross-contaminated and if we were in the back-country on a camping trip, professional medical aid might not be easily available even with an EpiPen handy. It is great to see people creating databases for themselves for product choices like this. Thanks BonzaOwl!

Rob Farley (@rob_farley | b)

Rob makes the astute observation that for a majority of his clients and personal needs, it is better to buy than to build a solution when you can afford it. Even smartphone apps have come a long way- it seems like there is an app for everything if you know what one is looking for. He has been in the data industry a long time and values both his own and his client’s time. Thanks Rob!

Steve Jones (@way0utwest | b)

And last but certainly not least, Steve talks about work in progress items across different fields like bucket lists, speaking events, and fitness data. Even SQL Saturday stuff! It seems like writing a lot of T-SQL to do data-entry is cumbersome from many of the posts. While I haven’t used Microsoft Access in ages, I do hear a bit of a resurgence in its use for simple client/server applications like hitting a SQL Database in the cloud. Steve and I have chatted about the MyLifeBits Microsoft Research project in the past and dealing with many disparate sources of data today for personal use is all over the place. Structured and unstructured data lives in many different places- and some of those places go out of business!

In Summary

I wanted to thank Steve for letting me host. This was my first time hosting. And Thank You everyone! Your participation has given me some great insights and ideas. –Todd

 

Posted in Uncategorized | 2 Comments

T-SQL Tuesday #113 – What Do YOU Use Databases For?

T_SQL_Tuesday

This month’s T-SQL Tuesday is about what do YOU use databases for in your personal life that you are willing to share?

We work with data all day and sometimes all night long. When we turn off work, some of us continue to use data and databases to enhance our professional growth and learning. I know on some days the last thing I want to look at after work is- another screen full of data. Kinda of like how some auto mechanics have poorly running cars. How can that be? Because they work on cars all day long and they are tired of working on cars- including their own.

But I’m curious- outside of work and learning, what do you personally use databases for? Tracking books you have, recipes, collections, etc? While it can be said using databases for personal use could be either overkill or a hammer in search of nails on the other hand, it is exactly what they are for- storing data.

Years ago I challenged my daughter- I would give her $100 to go through a book about databases, sit her in front of SQL Server Developer Edition, and we would build a database to track our VHS collection. (VHS, heh, yes I’m old. Now get off my lawn).

We used a book called The Manga Guide to Databases.

CoverBack

It was great except they used the non-ANSI syntax for joins…

non_ansi

We whiteboarded and created a simple database.

The experiment was a ton of typing. And linking to the movies in Wikipedia. And then capturing some information about personal family events based on date. BTW, she is now a film student in college.

More than twenty years ago I was into FileMaker Pro working for a defense contractor. We needed a way to store many pictures about launch site equipment. It was my first time trying to figure out: a) how to store pictures in a database b) how to store metadata about the pictures and c) how to search and retrieve the pictures. There was no auto-tagging, object recognition or any sophisticated technology to help me out. So I used what tools I knew about. The project was a success but took way more time than I would have liked.

I am into pictures and have been looking at ways to store and retrieve them for quite some time. Some things work and some things don’t work so well. On 06Apr19, I will be presenting in Colorado Springs, Colorado for SQL Saturday on Storing Images in a Database – Tips and Techniques.

As I learn more about Python I’ll be able to do more sophisticated things. Like the kind of things I couldn’t do twenty years ago. One of them I want to tackle is how to extract text out of images. Ever see people taking pictures during a presentation with their smartphones? Or see code in a YouTube video? How do you get the text of the code out? Most people I think would say they just re-type it. But what if you could use software to do that for you? So I’m fiddling with Python-tesseract.

So what do you use databases for in your personal life that you are willing to share? Blog about it. I’m curious to see the range of uses and even if you don’t use databases for personal stuff, what do you see as the future of organizing personal data?

All Good Things Must Come to an End

T-SQL Tuesday has the following rules:

  1. Publish your contribution on Tuesday, 09Apr19 between 00:00 UTC to 23:59 UTC.
  2. Include the T-SQL Tuesday Logo and have it link to this post.
  3. If your post doesn’t link back, please comment below with a link to your post.
  4. Tweet about your post using the #tsql2sday hashtag.
  5. Please consider hosting! If so, get in contact with Steve Jones ( blog | twitter ) to get your name on the list!
Posted in Uncategorized | 12 Comments

T-SQL Tuesday #112 “A New Cookie Jar”

T_SQL_Tuesday

This month’s T-SQL Tuesday is being hosted by @SOZDBA. Thanks Shane! BTW, T-SQL Tuesday builds upon the work of Adam Machanic and the continuing selfless efforts of Steve Jones. Thank guys!

My response to what is my reserve of strength as proposed in his post: “Dipping into your Cookie jar”

Too Many Links!!!

Guilty as charged. This topic activated A LOT of memories and thoughts for me. The benefits of getting older…

Background:
https://medium.com/the-mission/david-goggins-6-lessons-from-the-toughest-man-alive-cb2ea26fa18

Some definitions:
Bagfries (a term from software development. Exhausted but finding some bagfries.)
https://www.urbandictionary.com/define.php?term=bagfries

Running on fumes
https://www.urbandictionary.com/define.php?term=running%20on%20fumes

A cool documentary from the .dot com era about working under pressure doing software development:

Code Rush
https://en.wikipedia.org/wiki/Code_Rush
https://www.youtube.com/watch?v=4Q7FTjhvZ7Y

And if I could only see the future…

Styx – Crystal Ball
https://www.youtube.com/watch?v=9b-i0XknSnM

Yngwie J. Malmsteen – Crystal Ball
https://www.youtube.com/watch?v=aZSCt9w8DEc

And one of my heroes (fictional!) – Rocky Balboa
Rocky Balboa Speech – Motivation
https://www.youtube.com/watch?v=RC6sMFbL288

And I like how his elderly (!) trainer Mickey felt the need to keep on living by just watching Rocky train and fight so hard- it gave him hope.

Rocky Balboa – Motivational/Inspirational Speech To Son
https://www.youtube.com/watch?v=mk82j1jQw_8

Mental Visualization

Challenge is to see yourself doing it right, then you do it right. When it is done and when people are at their peak, it can account for up to nearly 25% physical training time. Meaning that is can help offset training time but not replace it. Being able to recall memories or knowing that you have overcome something hard in the past gives one fuel to press on. Nothing lasts forever…

Two books I like:

The Mental Athlete
https://www.amazon.com/Mental-Athlete-Kay-Porter/dp/0736046542/

10-Minute Toughness
https://www.amazon.com/gp/product/0071600639/

Now, all of these have to do with physical fitness. Using mental models to improve physical performance and looking into one’s past for examples.

What about sitting in front of a computer all day? Use these techniques to increase your typing speed, sure, but how can these models be used to make you a better data professional?

“…So tell me about a time when you had an accomplishment that can keep you going.”

Okay, one more motivational video.

The Matrix Reloaded – Morpheus Speech
https://www.youtube.com/watch?v=y5p_nM_xrww

I loved The Matrix series, still do. And The Animatrix was really good too. https://en.wikipedia.org/wiki/The_Animatrix

Now that affordable virtual reality (VR) is here, how can we build a new type of cookie jar? A new set of experiences we can fall back on when the going gets tough and we have no more bag fries or gas in the tank to keep going? When it seems like all hope is lost and we just can’t fight anymore?

A song from my youth. On the track from the movie Heavy Metal: Blue Oyster Cult – Veteran of the Psychic Wars: https://www.youtube.com/watch?v=7pFdkkjGiEs

This is where our future is heading. People facing outside battles and inside battles too. Scars on the inside. Nothing left to fight with after a long battle. Beyond exhausted and no resources to draw upon and no backups are anywhere in sight. Spent and all alone.

Now enter human likeness avatars. Of ourselves. Or something completely different. Which can be programmed to do the thing we want to be able to visualize ourselves being able to do well under extreme circumstances.

This clip is from SIGGRAPH last year. I have no idea what holds in store this year but I can’t wait to see what is revealed at The Game Developer Conference (GDC) next week as this kind of stuff is just going to get better and more affordable.

The actor Andy Serkis (from Star Wars, Lord of the Rings, and others) was digitally scanned and then rigged using software. Watch how digital Andy starts out as Osiris Black first then morphs into his digital self while reciting Shakespeare.

And a longer look at the tech & demoes. Remember this is ALL done in Unreal Engine 4!

So! Now just imagine an avatar of you (either real-life or something you can identify with) practicing the thing you want to get better at. Once the memories are deeply formed one will be able to re-call them in the midst of trials and tribulations. And make it easier to visualize success. This capability is probably a few years away but it is getting closer every day.

The cookie jar to dip into based on either our own successes in the real world or being able to jump into VR and see intensely memorable scenes based on what we are trying to do is going to be a new realm of human achievement. It will be new, scary as hell at times but it is coming.

How about an aged version of you, using a corpus of your voice sampled into artificial intelligence (AI) learning models (https://lyrebird.ai/), giving you advice today? Would you listen to your future self? How would that affect your behavior today? Or someone else important to you?

My existing cookie jar is knowing I have overcome problems and I trust my intuition, creativity, imagination, wisdom and experience. I have succeeded where others have failed, but not all of the time- I am still learning and growing.

My future cookie jar will continue along those lines but I am interested in a digital version of myself in a new way- using avatars to assist with mental visualization.

 

Posted in Uncategorized | 1 Comment

T-SQL Tuesday #111 “My Why”

T_SQL_Tuesday

This month’s T-SQL Tuesday is being hosted by @AndyLeonard. Thanks Andy!

My response to the question, “What is your Why? Why do you do what you do?”

I am a Christian. And I am unashamed.

I am also not a perfect example of a Christian. I am broken and imperfect. I fail and struggle every day about my faith. I sin, again and again and fall short of the Glory of God. But I keep working on it. Infinite Grace and Love and Mercy beyond my understanding. In ways that are visible and invisible. And I am forgiven. Everyday.

The Christian faith is not a once and done kinda thing. Just like decisions every day have consequences and repercussions, so too does an investment in whatever one calls the Bank of Faith.

My Why?

God, Family, Country.
Christian. Conservative.

My relationship with my Lord and Savior Jesus Christ, is something real and that I believe in every second of my life. Above all else. Period. Does that make my life perfect and beautiful?

No.

Far from it.

In fact, just the opposite.

If you have recently chosen to accept Jesus Christ as your Lord and Savior or have in the past, it will suck to be you (!). You will NOT be flying underneath the radar of spiritual attack. Sadly in fact, you just became a visible beacon for attack. Well, that is not something people want today! “I want to believe what I want to in my own little personalized, safe, self-created echo chamber without consequences, or pain or suffering!”

If one wants to play it safe, just let your defenses down, go with the flow, don’t fight against the tide, just believe in whatever brands/* isms/truth worldviews, etc. and not rock the boat, then the choice about faith is simple:

Do nothing.

Absolutely nothing.

Just let other forces in society push against the sails of your own ship. Surely they/groupthink knows what is best for you! If one has to do something, see what everybody else is doing first. Just give the people what they want and all will be great? Um, right?

No.

My Why?

I believe in lifting up those willing to help themselves. And to try to help those too who can’t help themselves. I have volunteered A LOT over the years. Unpaid. Unappreciated at times. Host meetings no one shows up at. But still putting your best face forward and doing it anyway not knowing what will become of it. Loss of time/money/effort and questioning if it is even worth it. And in hindsight, sometimes you get to see how you changed a life. That recently happened to me at our Denver SQL Server User Group meeting and it totally blew me away because it was completely unexpected.

My Why?

Because I genuinely love people in the same boat as me. No matter what circle that happens to be (SQL, start-ups, VR, Unreal Engine, pick a tech), I want to see people succeed and do well. If that drags me along, then so be it.

I once heard Zig Ziglar once say, “You will get all you want in life, if you help enough other people get what they want.”

At this stage of my life, I have had to deal with a lot of death. Death of family, friends, co-workers, and acquaintances. Murder, suicide, cancer, car accidents, heart attack, overdose, surgery, old age- the list goes on. Processing the emotions and thoughts is unique in each case and no, I am not looking forward to dealing with other deaths but I know it is coming. And it will come again and again, even unto my own death, at some point I hope and pray sometime into the far future. There is still so much work I want to do for many years to come.

My Why?

Because I don’t care about what people think about me (!). Sometimes I write hand-written notes and post them- some are even prayers on my part and I don’t know at times how they will be received. Will I offend somebody with my prayer for them? It is a risk I take and I do it anyway.

My faith is a integral part of me. I don’t proselytize at work but if someone starts a conversation about my faith I’ll talk to them. What you see is what you get. I don’t have a separate faith life apart from my work life. It is one and the same.

I know some people are going to drop me from social media like a lead balloon. I get it. It is expected. It breaks my heart to hear people in our SQL Family openly say stuff like, “I’m doing just fine without God”.

I have been told, that when I get to Heaven I will be surprised at the following: people who are there who I thought wouldn’t be there and those who are conspicuous by their absence- people who I thought would surely be there but who are not.

My Why?

Going back to Death. I have zero qualms or cares at this point in my life about sharing my faith. I have seen TOO MANY people I love die without knowing Jesus.

BTW, SQL Saturday Denver has been hosted a couple of times in the past at my home church. Do you know how many brochures, conversions, etc. I pushed on the people attending SQL Saturday?

Zero. Big Fat Zero. Because I believe in living the best life I can by example and let the Holy Spirit do the rest. If you ask me about Jesus, I will tell you. And pray for you.

My Why?

Money. Economic resources. If I ever become stupid rich, there is a ton of things I will do with my money. The cause of food, water, clothing, shelter here and abroad. So many people don’t know me that well. I believe God has blessed me with technical skills and creativity and imagination. I have a lot more to share for another time…

So part of My Why is to make money so I can give and spend it on things I believe in.

Two books that have helped shaped some of my recent thinking over the past several years about charities: When Helping Hurts and Toxic Charity.

My Why?

Because I am able, I can and I want to.

Posted in Uncategorized | 2 Comments

T-SQL Tuesday #110 – “Automate All the Things”

T_SQL_Tuesday

This month is being hosted by Garry Bargsley – thanks @gbargsley!

My response to this month’s topic is a personal one. The main two questions to answer are:

  1. What do you want to automate or what automation are you proud of completing?
  2. What is your go-to technology for automation?

This year my goal is to work on a double program, data science and artificial intelligence. I have chosen the Python path (or it chose me). So Python is becoming my go-to hammer in search of nails so I am all about, “Automate With Python All The Things”.

A book I am working through as I am learning Python is called, oddly enough, Automate the Boring Stuff With Python. LOTS of things I could automate that I had no idea about. Highly recommended.

Specifically, I am interested in Chapter 17, image manipulation.

Do You Take Pictures?

Please raise your hand if you take pictures. I see everyone’s hand went up. Yes, with digital cameras and smartphones, we take a plethora of pictures. Some would say too many! We don’t even take time to go back and look at all of them. Storage is so cheap we can be digital hoarders and not feel ashamed.

How Do You Backup Your Pictures?

Um, ya. About that. As data professionals we should know the importance of backups and having a restore strategy. Our backups should be local for immediate retrieval and off-site too. For personal use, backups should be off-site in some kind of cloud storage.

But how many of us regularly and with discipline and/or automation backup our pictures off of our smartphones, digital cameras, laptops and desktops? And is it a pure file-based backup or do you put pictures into a database?

Is Putting Pictures Into a Database Overkill?

Some would argue yes, that’s silly. Others would say no but how to do it? And what would the benefits be to storing images in a database?

Previously Done in R

I read this article a while back and worked through all of the examples. This was before I was bitten by Python. So I want to re-do all of this but do it in Python and store images in a database.

A Gift to My Wife

Chris and I have been married 25 years. As my gift to her, I am building an art gallery of images from the last 25 years. So far I am up to 300 images. Now, this isn’t a slide show. It is going to be a gallery built using Unreal Engine and deployed onto an Oculus Go.

But in order to properly size the images, they need to be a power of 2 as in 2048×2048. So! How do I fiddle with 300 images? Re-size, extract meta-data, store in a database, etc?

Enter Stage Right – Pillow and Scikit-image

Pillow (used in the book above) and scikit-image are both image processing packages.  Ever wonder how to count the number of marbles in a picture? Facial recognition software? These packages are amazing and they have a whole lot of really neat and cool capabilities. While some of the scientific stuff is over my head, I did pick up some things from this course- but I will need to re-take it again once I understand things better during the study of data science. Building Image Processing Applications Using scikit-image

SQL Saturday Topic

It shouldn’t be a surprise then that manipulating and storing images in a database is going to be my new SQL Saturday topic/theme for this year. It does sound like total overkill or using a cannon to shoot a fly. Some might say that is just good target practice. The technologies I am working with does sound like a lot and it is: Hekaton, FileTable, Python, Unreal Engine, and Virtual Reality for starters.

And before you ask, yes, I will be sharing all of my scripts and who knows, might even create my first community tool 🙂

 

Posted in Uncategorized | 1 Comment

2019 – The Year of Memory Recall

Nothing I create is ever perfect and I cannot wait any longer until it is- because that time will never come. Sometimes you just have to run with what you have.

So I’m going to focus on a single theme in my blog posts, presentations, and the like for 2019 – The Year of Memory Recall. All using SQL Server database(s) and technologies of course. And virtual reality. And Unreal Engine. And, and, and.

What follows is an incomplete rough draft of links and resources and comments. I had to post something in December…

NOTE!!! I am currently re-reading Total Recall and writing down even more notes, thoughts and updates since the book was published ten years ago when I first read it.

My goals and motivations will become clearer as time goes on. Memory recall is the act of recalling existing memories. The active act of doing so helps to strengthen and keep the memories alive. Memories are critical to shaping us as individuals and it is something we can control.

I am marching to the beat of a different drummer down the road less traveled. This journey will not be easy.

I am building a memory recall system which I’ll be using the rest of my life. If I live to be 100, I will be recalling memories I created in 2019 and earlier. A dual system- external memories and internal memories. Hopefully a system I can give to my kids- something tangible like my grandfather’s tool set and the memories I have of him wielding those tools building and fixing things. And versions of this system other people can use too for memory recall.

Some Sci-Fi devices (time to build a few?)

Star Trek
https://en.wikipedia.org/wiki/The_Lights_of_Zetar

Memory Alpha
https://en.wikipedia.org/wiki/Memory_Alpha

Uses a wiki encyclopedia. Created by fans, not by the creators of Star Trek.

Star Wars

Name of internal, real world tracking database (originally in FileMakerPro, one of my first database loves): Holocron; 55k entries. Clone Wars and Jedi Archives.

https://en.wikipedia.org/wiki/Star_Wars_expanded_to_other_media#Holocron_database_and_canonicity

http://starwars.wikia.com/wiki/Holocron

Jor-El; father of Superman; Fortress of Solitude; memory crystals
https://en.wikipedia.org/wiki/Fortress_of_Solitude

John Percival Hackworth; ~Nell; A Young Lady’s Illustrated Primer; nanotech; ractors; ractives
https://en.wikipedia.org/wiki/The_Diamond_Age

Aleph; Mona Lisa Overdrive
https://en.wikipedia.org/wiki/Mona_Lisa_Overdrive

Some Previous Real-World Attempts

Dymaxion Chronofile (analog scrapbook)
https://en.wikipedia.org/wiki/Dymaxion_Chronofile

Memex (analog but using electronics and hints of a digital version)
https://en.wikipedia.org/wiki/As_We_May_Think

Later Became Fully Digital
https://en.wikipedia.org/wiki/MyLifeBits

And used on the recommendation by the late computer scientist Jim Gray, “…you need to use a database like Microsoft SQL Server.”

Finally! Books!

Total Recall: How the E-Memory Revolution Will Change Everything (focuses on digital storage for EVERYTHING; of course, at a cost)
https://www.amazon.com/Total-Recall-Memory-Revolution-Everything/dp/0525951342

Going back to MyLifeBits- the problem and challenge I had was that MyLifeBits was an academic project. Absolutely nothing wrong with it, it is just that many academic projects do not easily lend themselves to be quickly implemented or have commercial applications. I have only seen screen shots of some ERD(s) and not an open source software or script library to pick up the pieces where the researcher(s) left off for others to follow. Sadly, many of the companies in the book are no longer in business.

MyLifeBits accurately predicted the falling cost of storage. For example, as of this writing, Amazon Web Services offers Glacier storage. $.004 per GB per month.
https://aws.amazon.com/glacier/

For life however… and pricing is continuing to drop, including Azure.

The problem? One could just use all the free storage, correct?

“Is digital storage a privilege or a right?”

Neither- it is still a cost, and it is NOT free. The cost however has fallen to ridiculously low levels and continues to drop. Again, if you don’t pay your bills, it WILL get deleted.

So unless you start a trust fund of some kind and choose to pay money forever (!) to keep it alive, it will die and be gone forever.

The Future of Looking Back (Developer Reference) – Microsoft Research
https://www.amazon.com/Future-Looking-Back-Developer-Reference/dp/0735658064

Why Isn’t Everyone Building Holocrons?

Because it is A LOT harder than writing and writing is already hard to do effectively. Heh, writing this blog post has taken more time and effort than I think it should.

For example, note taking. What if I told you that you had to create a video game every time you took notes? That’s crazy, no one wants to put that level of effort into note taking. But that is exactly how to create highly memorable content- a lot of work and concentration.

Software and security issues. Is the holocron a purely digital thing or is it a physical thing with digital storage? Or is it a purely analog storage device?

As a ~20 year SQL Server data professional, how can I use these ideas for myself? How can I build a holocron I can use time and time again to build new memories and re-visit old ones? How can I build something I can will as an heirloom or keepsake to my children or researchers? To build holocron(s) for my memoirs and for others.

Extreme Data Storage

Hybrid: Digital but stored using an analog medium DNA Storage

(1 exabyte per cubic millimeter!!!)
https://www.microsoft.com/en-us/research/project/dna-storage/

Analog Storage

Rosetta Project
https://en.wikipedia.org/wiki/Rosetta_Project
https://en.wikipedia.org/wiki/HD-Rosetta
http://norsam.com/rosetta.html

Conclusion

There is no conclusion. My journey to build holocrons for myself and others is just beginning in 2019. I am running with what I have. Using data science to sift and categorize and store things is a hint of where I want to go with some of these technologies.

 

 

 

 

 

 

Posted in Uncategorized | Leave a comment

T-SQL Tuesday #108 – Oculus Medium – Virtual Reality (VR) Play-Doh

This month’s topic is being hosted by @sqlmal – thanks Mal!

“So the challenge for this T-SQL Tuesday is – pick one thing you want to learn that is not SQL Server. Write down ways and means to learn it and add it as another skill to your resume. If you are already learning it or know it – explain how you got there and how it has helped you. Your experience may help many others looking for guidance on this.”

My topic is playing around with Oculus Medium.

A highly pixelated reference image imported into Medium, building a clustered index seek, exporting out from Medium and importing into Unreal Engine:

First, one will need an Oculus Rift headset, an adequately powered PC like a decent gaming rig, and some physical space to “dance” around in. Expect to spend $1000 to get into VR for the desktop and a lot more for a VR-ready laptop. This is a serious hobby for me. Second, one will need to buy the $29.99 software.

https://www.oculus.com/experiences/rift/1336762299669605/

What is it? From the Oculus website:

Sculpt, model, and play in virtual reality. Oculus Medium is an immersive VR experience that lets you sculpt, model, paint, and create tangible objects in a VR environment. Medium lets you create expressive works of art, whether you’re a total beginner, an aspiring creative, or a professional artist. Medium uses Touch controllers to enable intuitive hand gestures and movement for a natural, tactile experience.

Basic Alternative to the Expensive Stuff…

All of the desktop 3D modeling programs (Maya, 3ds Max, or the open source Blender which I like, or desktop sculpting like ZBrush or Mudbox) are either very expensive or really hard to learn or both. Think of them like a musical instrument- one has to practice every day in order to get any good. My goal is to be able to create things and import them into Unreal Engine (UE4) quickly. From UE4 I can then build or cook the project and install it on to something like OculusGo and in 2019, Oculus Quest.

https://www.oculus.com/quest/

Side note:

While Medium has not been announced for Quest, being able to take a VR headset with you anywhere and be able to create, ANYWHERE, is going to be simply incredible. I can’t wait to see what my amateur self and artists can do. Imagine an art studio that is with you all of the time. For creative people that is going to be the killer VR app- you and your studio. Being able to sculpt, draw or whatever in VR when the mood or idea hits you. Carrying multiple studios with a variety of projects in them with you. Did I mention you can cast your creations to a screen to show people what you are seeing in VR? Ya, this is going to be awesome when Quest comes out. And yes, I am thinking of how to do Quest presentations/performances as topics for SQL Server. Goodbye PowerPoint.

Getting Started

I didn’t want to wait years to create things. I wanted to play now. Much like Play-Doh or Legos one just needs to learn the basics and jump in. One uses this VR software standing up so be prepared to be standing for a while and try not to get tangled up in the headset cables or knock things over. What one creates in Oculus Medium are called sculpts.

How has it helped me?

Escape. A chance to experience Flow for hours on end. Get out of my comfort zone. Something that doesn’t involve coding.
https://en.wikipedia.org/wiki/Flow_(psychology)

Flow: The Psychology of Optimal Experience
https://www.amazon.com/gp/product/0061339202/

Flow. Ideation. Creation. Those are the goals of playing in Medium.

I have ideas for all kinds of inventions. A soft spot in my heart is about food, water, clothing, and shelter, with drinkable water being most important for survival and development. Topics for other time but being able to try and build things in my mind’s eye inexpensively is an expression of my creativity and imagination. Medium is a constructive outlet for me.

How to Learn Medium

Recently Oculus has updated Medium to 2.1 and has made a lot of great, free tutorials. Start with the tutorials INSIDE the software. Then, try these out. Even if you don’t have a VR set-up, watch these anyway to get an idea of what one can do with the software:

https://www.oculus.com/medium/learn/

It can take a while to learn the basics. Watch a short tutorial then try to duplicate what the artist is doing. Watch and try again. And again.

Note this is the first version of the genre of sculpting software using VR so it can be buggy at times. SAVE OFTEN!!!

The smallest resolution is 1 cm. Which just happens to match the smallest unit in Unreal Engine, the Unreal Unit (UU).

So if one is trying to build a fine mechanical watch using digital clay, prepare to be disappointed. One can “zoom” in super-close to the clay and do some fine-detail work but realize this is still clay, not micro precision molecular assembly like nanotechnology. Not yet anyway.

I have found the triangle/polygon count to be ridiculously high for imported sculpts. There are some tools to help with that (reduce number of triangles while maintaining visual fidelity on final exported .obj file) but I have not learned how to use those yet. Here is one I read about, MeshLab:

https://en.wikipedia.org/wiki/MeshLab
http://www.meshlab.net/

As far as exporting sculpts into UE4, I have found exporting as .obj with textures at the 2048 setting will generate good WYSIWYG. Import into UE4 then re-size as needed. Some of the colors can be a little off but I have been happy with the results. Please note VR is a resource intensive application- only have one thing running at a time. This means don’t have UE4 running while trying to use Medium and a web browser with a tutorial. Just one at a time. And yes, one will be re-starting your desktop often.

Just so happens that on the day I post this, I am also giving a presentation on using Oculus Medium at the Unreal Engine Meetup in Denver which I run. Starting a new user group is hard…

Hobbies on Resume

At what point does one add skills developed as a hobby and put them on a resume?
What I have heard is only if professionally relevant to a job one is applying for.

I don’t consider myself qualified for this position but it does have things I am interested in:

https://www.unrealengine.com/en-US/blog/jobs-in-unreal-engine—technical-artist

My dream job would be working with data and virtual reality. We’ll see what next year has in store.

I hope this post helps get some creative juices flowing. Feel free to hit me up on twitter:

@toddkleinhans

Thanks!

–Todd Kleinhans

Posted in Uncategorized | Leave a comment