9/28/2016

How Hibernate Almost Ruined My Career


Hibernate Almost Ruined My Career - Sleeping beauty.
Imagine that you are a Java developer, and you’re about to start your next big project. You need to make the fundamental decisions that will stick with you for the rest of the project. You want to pick the best object-oriented abstraction of your flexible data model because you do not want to deal with plain SQL. You want to support all kinds of data, and ideally, support all sorts of databases.

The obvious answer is to just use Hibernate, right? 90% of Java developers would agree with you, but does that make it the right decision?

Let’s take a look at what can go wrong if you blindly use Hibernate just because it’s the accepted standard.

Consider Monica, a Java developer. Monica has recently been promoted to the role of architect and is now responsible for laying out the technology stack for a new product at her company. She knows that in the Java world there is only one good tool for handling database communication: Hibernate. Hibernate is a well known and supported JPA standard. However, it is always a good idea to check a few things before starting a project. Fortunately, her colleague, Ben, knows the right guy.

4 Years Ago, Hibernate Sounds Like A Silver Bullet


  • Ben - Hello Monica, I’d like to introduce John. He’s a Hibernate expert, and he’s going to help you.
  • Monica - Hey John, glad you found some time for me. So, we are building our Next Big Thing, you know. We are planning on becoming the next Facebook or Google. Busy days. It’s going to be huge. Absolutely fantastic! Everybody is so excited! I’ve been promoted to the role of an architect, so now I have to select the stack we will be using. The only missing part is persistence …
  • John - Hibernate!
  • Monica - Yes! Exactly! Just what I was thinking! It seems like a perfect match and the real deal for us. A true enterprise solution for a true enterprise problem, proven by the market and with a long history. I’ve heard so many positive experiences with it. However, I have an issue with one of our teammates; he is totally against it. He knows a lot about databases, and he’s afraid of adding another layer between our application and the database. He is super smart, and I need some really good arguments to convince him this is a good decision. Can you help me with that?
  • John - Of course! I will be glad to. Hibernate is, indeed, an outstanding tool. It’s widely used in big, true enterprise solutions, like banks. You can’t go wrong with it. Think persistence: Pick Hibernate. If you are writing in Java, this is absolutely the right choice, plus you have ports for other languages. See how many job descriptions require it!
  • Monica - I absolutely agree! I have the same feelings about it. In a previous project, we were using mostly SQL via plain old JDBC. Ridiculous! I know! But, here’s the thing: We have really smart SQL guys in the team and when they saw SQL generated by Hibernate they got nervous. It seemed ugly and unreadable; will this be a problem in future?
  • John - Look. DBA guys have a different perspective. They are afraid of Hibernate because it seems to replace their role in the project. Moreover, databases have built-in query optimizers so you don’t need to worry how those queries will actually look. The database will optimize it for you. It’s all about rapid development, which SQL can’t do.
  • Monica - Really?! No longer dealing with SQL? Amazing! Last time a DBA spent weeks trying to optimize some queries. Weeks! Oh, I feel so embarrassed telling you this, but did you know that we were using … stored procedures (laughing). Oh, it was such a mess. Can you believe the project is still using it? I feel so sorry for people out there. They still have to write this tedious code over and over again. I wonder if it’s still a Java or SQL project?
  • John - That’s exactly the difference between an object-oriented approach and the relational one. It’s a so-called object-oriented impedance mismatch. Hibernate can close this gap. Developers can focus on building business logic. Push features make stakeholders and the entire management happy. Do the things that matter most: Business! Lots of boilerplate code will disappear, and you’ll have a magical, invisible, but reliable, connection between the logic and the data.
  • Monica - Mutual cooperation. Full synergy. Like the database was part of the language from the very beginning. I am so happy I get to be a leader of this technological leap of faith.
    It’s like warp speed in the software trek.
  • John - Yep! You’ve got it!
  • Monica - Oh gosh, I’m so excited! Thank you, John! I’m ready!

3 Years Ago, Growing Pains With Non-Flexible Solutions


  • Monica - Hey John, remember the project we spoke about last year?
  • John - Sure. How’s it going?
  • Monica - We are going to production soon. Everything is fine, but some questions have popped up.
  • John - Sure, hit me.
  • Monica - Well, we can no longer generate our database schema from scratch. What’s the best way to support schema changes without losing data?
  • John - Well, first, Hibernate isn’t intended to be used as production migration tool. Use something like FlywayDB or Liquibase. It’s pretty simple. You write down migration scripts, then you update the entity model along with the Hibernate mappings, so it keeps in sync with the actual database structure.
  • Monica - Hmm, I see. We were using just plain SQL migration in the previous project.
  • John - That’s fine too. As long as you keep the entity model and schema in sync, do it how you like.
  • Monica - I see. There’s another thing. We’re always struggling with lazy/eager fetching problems. At one point, we decided to do everything eagerly, but it seems suboptimal, and besides, sometimes it’s not possible to access some fields because there is no session, or something like that. Is that normal?
  • John - You need to learn more about Hibernate. Mapping from the database is not straightforward. Basically, there are multiple ways of doing it. You just need to pick a way that works for you. Lazy fetching gives you the ability to load those objects on demand, but you need to operate within an active session.
  • Monica - We’re still struggling with which database engine to use for the final deployment.
    I thoughtHibernate was portable, but we have some native queries that use some MS SQL magic, and we’d actually like to go with MySQL in the production.
  • John - Hibernate gives you flexibility as long as you are using detached criteria or HQL;
    any native queries will just bind your solution to the database.
  • Monica - Seems like we have to stick to the MS SQL then. Last question: My teammate said that there is no “limit” keyword in HQL. I thought he was joking, but I couldn’t find it either. Sorry for the stupid question…
  • John - Indeed, there is no “limit” keyword in HQL. You can control this via query object since it’s database-vendor specific.
  • Monica - Seems weird that all the other elements are in HQL. Nevermind. Thanks for your time!

2 Years Ago, We’re Now Hacking Together Solutions In SQL Again


  • Monica - John, at the beginning we weren’t going to deal with SQL, but now it seems like we have to. Our needs are growing, and it seems like there’s no way around it. It feels wrong, but we’ve started using SQL again on a daily basis.
  • John - Well, it’s not wrong. You didn’t have to focus on the database at the very beginning. However, as the project grows, it’s good to use SQL and work on the performance optimization.
  • Monica - Sometimes we spend days looking for errors. It seems like we have to analyze Hibernate-generated SQL because we have no idea why it’s not working as expected and it’s producing unexpected results. We hit some problems that are well known in the Hibernate bug tracker. Additionally, it’s hard to write proper migrations while keeping the entity model in sync. It’s time-consuming since we need to learn a lot about Hibernate internals and predict how it’ll work.
  • John - There’s always a learning curve. You don’t have to write much, but you do need to know how it works.
  • Monica - Working with bigger datasets is also annoying. Recently, we did a massive import to the database, and it was painfully slow. Then we found out that we had to clear the session to make it faster. Even so, it’s still significantly slower, so we decided to rewrite it as plain SQL statements. What’s funny is that writing plain SQL was actually the fastest way of doing it, so we decided to do it as our last option.
  • John - Import is not an object-oriented process. Hibernate focuses on object-oriented design. Remember that you can always use native queries.
  • Monica - Can you help me understand how Hibernate cache works? I just don’t get it. There are some first/second level caches. What is this all about?
  • John - Sure. It’s a so-called transaction-level cache of persistent data. It’s possible to configure a cluster or JVM-level cache on a class-by-class and collection-by-collection basis. You can even plug in a clustered cache. But remember that caches aren’t aware of any changes made to the persistent store by another application. They can, however, be configured to delete expired cached data regularly.
  • Monica - Sorry, think I’m having a bad day. Can you explain this a bit more?
  • John - Sure. Whenever you pass an object to save, update, saveOrUpdate, or retrieve it via load,get, list, iterate or scroll, that object is added to the internal cache of the session. You can also remove the object and its collections from the first-level cache.
  • Monica - Er…
  • John - Additionally, you can control cache modes. You can use normal mode to read and write items to the second-level cache. Use get mode to read from the second level but you can’t write back. Use put, which is same as get but you can’t read from the second level. You can also userefresh mode, which is going to write to the second level, but not read from it and bypass the use minimal puts property, forcing a refresh of the second-level cache for all items read from the database.
  • Monica - I see. Ok. Let me think about this. Oh, it’s late, I need to go. Thanks for your time!
  • John - You’re welcome!

2 Weeks Ago, Giving Up On Hibernate


  • Monica - John, I thought we were entering a new era of software development. I thought we were doing a light-year’s jump. But, after four years, it seems like we are still dealing with all the same problems, only from a different angle. I had to learn Hibernate architecture, configuration, logging, naming strategies, tuplizers, entity name resolvers, enhanced identifier generators, identifier generator optimization, union-subclasses, XDoclet markup, bidirectional associations with indexed collections, ternary associations, idbag, mixing implicit polymorphism with other inheritance mappings, replicating object between two different datastores, detached objects and automatic versioning, connection release modes, stateless session interface, taxonomy of collection persistence, cache levels, lazy or eager fetching and many, many more. Even with everything I know, it seems like we’ve failed badly. It’s a software fiasco! Ultimate failure! Disaster! Armageddon!
  • John - Wait! What happened?
  • Monica - We’ve reached a dead end. Our application performance is ridiculously slow! To get a report, we have to wait two days! Two days to actually generate a dashboard for a customer. It means every day we have to increase our calculation tail, while our dashboard gets more and more outdated. Our DBA expert has been working two months to optimize some queries, while our database structure is a complete mess. There are developers supporting him, but the problem is the DBA is thinking in SQL, and the developers are spending days trying to translate this into detached criteria or HQL format. We are trying to use native SQL as much as possible since performance is crucial at the moment. Anyway, we can’t do much since the database schema just seems to be wrong. It felt right from the object-oriented perspective, but it seems ridiculous from the relational one. I’m asking myself: How has this happened? The developers are telling us changing the entities structure is going to be a massive effort, so we can’t afford that. I remember in the previous project it was a mess, but we never wound up at such a critical point. We were able to write an entirely different application to work with the data. Now, it’s risky to modify those generated tables since it’s really hard to make sure the entity model will always behave properly. And this isn’t even the worst part! To increase performance, we have to solve not only database issues, but also issues with the entire layer between our database and the application. It’s overwhelming! We have these new guys, you know, consultants. They are trying to extract data, put it into some other storage and then perform calculations from the outside. It’s all taking too much time!
  • John - I don’t know what to say.
  • Monica - You see John; I don’t want to blame you. I picked Hibernate to solve all these problems, but now I’ve learned it’s not a silver bullet. The damage has been done, and it’s irreversible. Actually, I would like to ask you something: I spent the last four years of my career dealing with Hibernate stuff. It seems I do not have a future at my current company. Can you help me?

Today, So What’s The Lesson Learned?


  • John - Hey, Peter, let me introduce Monica.
  • Peter - Hey, Monica! We’re building our new next big thing you know. It is going to be huge! We want to be like Uber! Do you know maybe how persistence…
  • Monica - Not Hibernate!

Wrap Up


Monica is a Hibernate expert. However, Hibernate in this instance was a wrong decision. The moment she discovered that her solution turned into a bigger problem than the original, it was the largest threat to the whole project.

Data is the central purpose of the application and, like it or not, affects the entire architecture. As we learned from the story, do not use Hibernate just because your Java application is using a database or because of social proof. Pick a solution that embraces flexibility. There are plenty of options for robust JDBC wrappers, such as JdbcTemplate or Fluent JDBC Wrapper. Alternatively, there are other powerful solutions, such as jOOQ.

See what people say about this article on the Toptal or Reddit page.

3/08/2016

Custom interactive CSS/HTML tooltips with ggplot, shiny and R (ggvis like).


Working on data set visualization with R and shiny for Facebook data insights, we've faced a serious problem with some, one would say "simple feature".

"Hovering on plot data, shall display popover (tooltip) with the user picture, active link, short description and some other data".

First we've tried ggvis library. It was very promising and the tooltips worked like a charm.
However, very soon we've found out that library is not mature enough and leads to problems with plot interactions i.e.  zooming - to get rid of marque artifacts we would have to patch the library with some custom JS and other "fixes". This way looked "quick and dirty" so we've decided to explore the possibilities of well known and mature ggplot. If it's possible to display the data we need in "panel", let's fetch the panel with needed HTML data and play with it's position and look. The final solution is as follows:

Tooltips are managed by combination of shiny+ggplot hover functionality and CSS styles.
By setting hover argument of 'plotOutput' we can access  hover data from the server side, as an ordinary input.

Hover input is a list with:
  • position of cursor ON the image; 
  • domain - that is values of variables at the plotting area edges; 
  • range - that is position of plotting area edges in pixels relative to whole image element.

Additionally for ggplot mappings are returned.

To create tooltip first we need to identify position of the cursor inside the image element. We do it by calculating distances from left and top edge of image element from hover data. Then we create tooltip, in this app it is 'wellPanel' with required informations inside, and 'position' property set  to 'absolute' and set 'left' and 'top' properties to calculated values.

However, 'absolute' position is defined as relative to the nearest positioned ancestor. Because we want to position tooltip inside the image, we need to put both 'plotOutput' with image and 'uiOutput' with tooltip content inside additional 'div' element with 'position' property set to 'relative'.

We don't set top, left etc. for this element, so the actual position of the image doesn't change - it's edges are identical as previously, so we can use 'div' (for positioning tooltip) as substitute for image.

You can find full, working, simplified code snippet here: https://gitlab.com/snippets/16220

Special thanks to Bartek Chroł for help with developing this idea!!

ggplot with fully customised tooltip