Combining Data Sources

Data combining is the method of joining data from various sources to produce a single, unique dataset for analysis and presentation.

**Why is it important to know how to combine data sources and how it is going to help you when working with data?**

1. **It improves accuracy and completeness of your dataset :** When you have multiple sets of data, it is important to make sure that they are accurate and complete. This can be done by merging the sets together. By merging the sets, you combine the information from each set into one. This helps ensure that all of the information is accurate and complete. It also ensures that there are no duplicates in the set.
2. **Combining data sources helps paint a better picture of what's going on in the company:** When businesses merge data from multiple sources, they can gain a more accurate picture of what is happening within their company. This is because different data can provide different perspectives on the same issue. For example, sales data can be used to measure how well a business is performing, while customer service data can be used to track customer satisfaction levels. By combining information from multiple categories, organizations can gain a complete understanding of their overall performance.
3. **Improves decision making:** By combining data, businesses can gain a better understanding of what is going on in the company. This allows them to make better decisions based on real-world information rather than relying on estimates or assumptions. Additionally, businesses can use merged data to identify trends and patterns that may not have been visible before. This can help them improve their operations by identifying areas where they need to make changes or improvements.

### **TYPES OF DATA COMBINING**

When combining different data sources, there are two ways to connect the outsource data to the existing table:

1. Appending Data Sources
2. Merging Data Sources

Appending is attaching one or more rows from one table to another whereas merging is attaching one or more columns from one table to another

#### **Appending Data Sources**

Let us now go through the different examples of how an appending on two different data sources look like.

**Example 1:** There are two distinct data sources, designated as DF1 and DF2, as seen in the image below. We now obtain the resulting data source DF by appending DF2 to DF1. As with the column Grade, the rows of Student Name in DF2 are being added to the rows of Student Name in DF1.

**Example 2:** There are two distinct data sources, designated as DF1 and DF2, as seen in the graphic below. We now obtain the resulting data source DF by appending DF2 to DF1. The columns Grade and Student Name in DF2 and DF1 are both being expanded by the rows of Student Name in DF2, respectively. However, as the Subject column for the DF2 has no values, "null" values are printed in their place in the new data source DF.

**Example 3:** Example 3 is similar to Example 2, but DF1 does not have a Year column. There are two distinct data sources, designated as DF1 and DF2, as seen in the graphic below. We now obtain the resulting data source DF by appending DF2 to DF1. The columns Grade and Student Name in DF2 and DF1 are both being expanded by the rows of Student Name in DF2, respectively. However, since the Year column for the DF1 does not include any values, "null" values are written

**Example 4:** Let's now switch to a trickier adding method. Both of the data sources lack the columns in question here. There are two distinct data sources, designated as DF1 and DF2, as seen in the graphic below. We now obtain the resulting data source DF by appending DF2 to DF1. The columns Grade and Student Name in DF2 and DF1 are both being expanded by the rows of Student Name in DF2, respectively. However, since the Year column for the DF1 does not include any values, "null" values are written in their place in the new data source DF. Additionally, there are "null" values written in DF but no values in the Year column for DF1.

**Exercise on Appending Data Sources**

Let's learn how to append with the Power BI tool now that we are familiar with the appending notion. We have two data sets, one of which contains data on states with a population under 5 million and the other with a population over 5 million. These two data sources must now be combined into one. As you do the following:

1) Load the ExampleData.xlsx file from the Canvas (select/load each sheet in the file).

2) Go to the data view now, and click the "Transform data" button on the main tab. When you click on it, a new window with the data can be seen and edited. The new window is depicted in the image below.

3) Double-check to see that the Population > 5M table is selected. Choose Append Queries as new by clicking on Append Queries on the home tab right now (you can also select Append Queries but it appends in the same table).

4) Now change "Append 1" to "Population" in the new table that appears at the bottom.

There you go! We have now mastered the appending technique for combining data from two sources.

#### **Merging Data Sources**

Now that we’ve discussed appending data, we will show you how to merge two data sets. Starting off, we will get our data saved into an excel file. Once you have the data saved, you will then open up Power BI. When the screen opens, you will then click on the “Get Data” tab on the top left side of the main screen. You will then click on the data source your file is saved under and upload it into Power BI.

- Note that you are able to use any of the data sources provided.

Once you’ve clicked on the data you want to upload, you will need to select the tabs of data you would like to merge together.

Once you’ve selected your data, you will then click the yellow “load” button on the bottom right-hand side. Now that we have the data we need uploaded into Power BI, we will begin Merging them together. The next step is to click on transform data:

You will then be brought to the screen labeled below where you will click on merge queries (Top right-hand side of the screen).

Upon selecting Merge Queries, the tab listed below will come up. The down arrow listed below the first data set is the secondary set you would like to merge. Upon hitting that down arrow, you need to select the second data set you’d like to transform. Now that both data sets appear, you will need to click on the columns you would like to match. For this example, we are matching Per state Population with Median Income per State. In this case we need to select the State or Territory in both data sets to ensure Population and Income both matchup together. You will than see a green checkmark at the bottom showing you how many row matches you will have upon merging them. Now that we have everything selected, we will select the yellow “OK” button located on the bottom right-hand side of the screen.

After merging the data, the picture listed below will appear. In order to show the values of the far-left column, you will need to hit the double arrow icon shown below. You will than only select the column you are trying to generate values from.

- Note: Every column will be initially selected after hitting the double arrow icon. You will simply need to deselect all columns and only click on the one you’re trying to generate values from.

Lastly, the final step is to click on the “Close and Apply” Icon listed on the top right-hand side of the screen.

Now your Two data sets have successfully been Merged!

##### Join Methods/Fuzzy Matching

To better understand how your data sets are being combined, its important to know the different kinds of joins. Because of the underlying code, Power BI shares join methods with SQL (See image). First, the default join is called a Left Join. Using a left join would keep every item from the first data set (A) and add a column with the information the program was able to match. This means if there were any lines in Set A they were unable to match, they would remain in the final product. The lines Power BI is unable to match would be filled with “Null.” But what if you don’t want to have any “null” values with unmatched values? You can use an Inner Join. With an inner join, Power BI is only keeping the lines they were able to match exactly. Overall, when determining what join to use, it’s important to take time and think about how the different methods would affect your data sets. Different scenarios call for different joins, so just be aware of how they interact with your data so that you don’t accidentally include/exclude anything.

Another tool that may be applicable is Fuzzy Matching. You can find fuzzy matching as a dropdown near the bottom of the merge menu. By opening that menu, you can see additional tools that allow you to customize how “specific” Power BI is when finding matchings. This can be useful when different datasets may not have exact matches within their data. By using a different “Similarity Threshold” you can better customize how specific Power BI is. You can also have the program ignore/not ignore cases and by combining by automatically combining separated words.

##### Merge w/ Multiple Columns

There may be situations when you need to have Power BI analyze multiple columns to make a correct match. A great example of this dilemma is names. Some names can be common, thus in some data sets there may be multiple instances of a single name, but they represent different people. You can see this in the data sets show left and below. While There are two Alans, one is Alan Sherman and the other is Alan Ajroututu. If we were to do a standard join (see above) with just the first names, we would run into problems. When a line would correspond with Alan Sherman, Power BI may instead use Alan Sherman. Because of this, we must tell Power BI to not only match by first names, but also last names. As long as both data sets have the required columns, this a pretty simple step within Power BI.

First, you go to the previously mentioned merge menu. From here you can select the first columns you want to match. In this example, I clicked the two “First Names” columns. From here you must hold the “ctrl” button and select the next two columns you want to match, “Last Names.” When you do this, you can see that Power BI now numbers the multiple columns (See left). Make sure that the numbers correspond to the correct columns between the two data sets.  After confirming the correct columns, you can see, using the check on the bottom, that Power BI was able to match all 39 rows. Once you select “OK,” you can see that Power BI now correctly merged the two data sets by taking BOTH first and last names into consideration when finding matches. While this may not be applicable in every scenario, its important to be aware of this feature so that you don’t accidentally exclude any important data due to an incorrect merge.

### **CONCLUSION**

In a nutshell, by combining data from different sources, businesses can gain a more comprehensive understanding of their customers, products, and operations. This, in turn, can help them make more informed decisions and improve their performance.

### **REFERENCES &** **LINKS**

1. (Combining Data Sources)https://thetravelmanuel.com/the-benefits-of-data-merging/#:~:text=By%20combining%20data%20from%20different,decisions%20and%20improve%20their%20performance
2. M, L. (2022, July 5). *The Benefits of Data Merging*. The Travel Manuel. https://thetravelmanuel.com/the-benefits-of-data-merging/
3. (Appending Data Sources)https://learn.microsoft.com/en-us/power-query/append-queries
4. (Merging Data Sources)https://learn.microsoft.com/en-us/power-bi/connect-data/desktop-shape-and-combine-data

### **DISCLAIMER**

*This wiki page was created by students during a class project on November 30 2022. If you need help understanding anything else, please feel free to use the links provided to learn more. Thank you!*