Как из json вытащить нужные данные python

Parsing JSON files using Python

Take a look at an easy guide on how to use Python to read JSON files.

Death touches each one of us, but it simply is not a pleasant topic to talk about! But, contrary to that, in today’s world of data and analytics, we should be discussing death because using the mind-boggling amount of data that we generate everyday, we can look at the topic of death with a positive lens. What if medical practitioners could reduce the death count using data and help us live happily and enjoy our life? Could you imagine the impact it can have on all of us?

Medical deaths due to diseases have a higher chance of prevention and cure if treated correctly on time. Accurate information about different diseases affecting majority of the people in a given area would be invaluable for medical practitioners. They could advise people on the preventive measures, mark out a plan to research on vaccines targeted to prevent those diseases as well as formulate new drugs to cure patients. All of you would agree that in today’s world of data explosion, collecting accurate data would not be a problem!

In this article, we will investigate a similar data set which tells us about the leading causes of death in New York city since 2007. Year-wise trends from 2007 can help measure the outcomes of new medicines/vaccines and its effect on the death rate. It will also help us shortlist diseases which are becoming more prominent and need urgent attention. There are quite a few questions we could answer using this data set, including:

• Does the death count vary based on sex?
• What are the major cause of deaths per sex?
• What are the major causes of death race-wise?
• What is the year-wise pattern of causes of death?
• What are the major causes of death in young people?

Enough of talking! Now, let us get started with the actual work. The data we have is a JSON file. What exactly is JSON and how are we going to parse it in Python? Let us peek at the beginning of our JSON file:

Yup, this looks confusing, but we will decode it using the below road map:

• Understanding JSON
• Reading from a JSON file and extracting it in a Data Frame
• Cleaning the data
• Analyzing data to get answers to our questions.

Understanding JSON

What is JSON? Why do we use JSON?

Before we jump into the actual definition of JSON, let us look at an example. Let us say that two people, one person speaking Hindi only and the other Chinese only, want to communicate with each other. How can they talk if they don’t understand each others languages? They would need a translator! But, wouldn’t it be easier and faster if they knew a common language? In the same way, say suppose JavaScript and Python want to communicate. They don’t know each others language, but if they know one common language, they can send and read messages. Also, they cannot read a dump of text messages, they need formatted messages with some grammar and syntax. A simple language independent format of organized messages which both languages can read is JSON. Both JavaScript and Python have built-in JSON libraries to convert JSON into their respective programming languages which they understand.

JSON objects are human-readable lists of name/value pairs. JSON data looks much like a dictionary would in Python, key:value pairs. This format encodes data structures like lists and dictionaries as strings to ensure that machines can read them easily. It can have nested name/value pairs as seen below for ‘address’.

JSON is easier to parse, quicker to read and write. JSON syntax is based on JavaScript, but is in text format. Now that we know the basics of JSON and its structure, let us jump into our source JSON and read it in Python.

Reading from a JSON File and Extracting it in a Data Frame

Exploring the JSON file: Python comes with a built-in package called json for encoding and decoding JSON data and we will use the json.load function to load the file.

Even if Python has an in-built library, we still need to know how to find the data we need. In our JSON file, you will see two top level keys:

• meta: This usually refers to information about the data itself. This key has other nested key: view.
• data: This is the second key containing the actual data. (You can search this key in the big JSON file.)

When you explore the data key in the file, you will see a list-of-lists containing the actual data, but without column names as shown below.

So, before we explore how to extract the data, let us see how we can access the column names.

Getting header information (column names): Keeping in mind the key:value syntax, as you scroll down to the meta.view.columns key in the JSON file, you can see many items nested in the values. Items have key:value pairs starting with ‘id’ and ending with ‘flags’ keys.

If you observe the above code snippet, you can see that the ‘fieldName’ key has values like sid, id, position, created_at. As you open and scroll down the JSON file, you will see other values like year, leading_cause, sex, etc. Looking at these values, we can say that ‘fieldName’ is the relevant key containing information about the column names that we need. We just have to extract the ‘fieldName’ key from each item chunk in ‘columns’. Let us look at the code for extracting the column names.

We will use the ijson library which iteratively parses the json file instead of reading it all in at once. This is slower than directly reading the whole file in, but it enables us to work with large files that can’t fit in memory.

After opening the file, we will use the ijson.items() method to extract a list from the path of meta.view.columns from the JSON file. Here, meta.view.columns.item indicates that we will extract each individual item in the meta.view.columns list. The ‘items’ method will return a generator and we can use the list method to convert it into a Python list. The first item of the list ‘columns’ looks as given below:

Each item in ‘columns’ list is a dictionary that contains information about each column. You can double check this is the above snippet of the JSON file. As discussed, ‘fieldName’ key is important for us. Let us look at it in detail.

Perfect, we were correct about the ‘fieldName’ key and we can see that it has the columns that we need! Now, we can move to extracting the data for these columns.

Extracting Data: Let us select the columns that we would need for our analysis.

As discussed above, the top-level key ‘data’ has the actual data locked in a list-of-lists. We can now use the columns extracted in ‘good_columns’ to fetch data only for columns we need.

• The ijson.items() method will return each item in the data key using data.item path.
• The first for loop will loop through each item in data key giving us a row-wise list.
• Once we have the row-data, we now want to extract data only for the ‘good_columns’. Hence, we will use another for loop to go through each column data in the selected row. We will find the position for our selected columns using: column_names.index(item) where item will be from ‘good_columns’.
• row[column_names.index(item)] will give us the data for the ‘good_columns’ in each row, which we will append to our list ‘selected row’. In this way, we will select data for each columns of ‘good_columns’ in that row.
• Once this is done, we will follow the same procedure for each row and append our final data which will be a list-of-lists to ‘data’. We will then convert it into a data-frame for easier analysis.

Awesome! Our data from the JSON file is ready in a data frame for analysis.

We can now move on to data cleaning and analysis to answer our questions.

Cleaning the Data

This is an important step before we go ahead with our analysis as clean data (without null values, correct text, correct data types) is a prerequisite for any calculations and inferences. We can work on the below points. For details you can refer to the notebook.

• Change None, null and ‘.’ values in ‘deaths’ column to 0 and convert it to numeric.
• ‘Sex’ column has different values: ‘F’ and ‘Female’, ‘M’ and ‘Male’. We will convert all values to ‘Female’ and ‘Male’.
• Change ‘.’ values in ‘death_rate’ column to 0 and convert it to numeric.
• ‘leading_cause’ is an important column as it is giving us important information about the death, but it has some numbers which we will remove to clean the column values.

Analyzing Data to get Answers to our Questions

For detailed analysis, you can refer the notebook.

• Does the death count vary based on sex?

We can see that the number of deaths in males and females are almost equal with female count slightly higher.

• Are number of deaths for each sex varying across years?

1. Both male and female death counts from 2007–2016 have been more than 25000 each year. Female death count was highest in 2008, while male death count was in 2016.
2. There is a dip in the female death count from 2009–2014, but after that it has increased again. Male death count has increased consistently from 2009 on wards.

• What are the major cause of deaths per sex?

1. Male and Female leading causes of death are diseases of heart, malignant cancer. Number of female deaths due to these two causes has been higher than males. High counts maybe due to improper lifestyle of junk food, chemical exposure, habits, overall lifestyle and stress.
2. Assault and Parkinson’s disease are affecting males only. Males have a higher chance of death due to accidents except drug poisoning, chronic liver disease, HIV, suicide, mental disorders — accidental poisoning.
3. Females have a higher chance of dying due to Alzheimer’s disease, cerebrovascular disease, respiratory diseases, hypertension and renal diseases, influenza and pneumonia, nephritis, septicemia.

• What are the major causes of death race-wise?

Along with diseases of heart, cancer, diabetes, influenza and pneumonia, some of the prominent causes of death for different races are:

Парсинг JSON в Python

Модуль JSON входит в стандартную библиотеку Python и является эффективным средством взаимодействия с JavaScript Object Notation (именно так расшифровывается JSON). Функции этого модуля дают возможность разработчику кодировать и декодировать информацию при работе с различными JSON-объектами. Всё это существенно упрощает создание веб-приложений в Python.

Пару слов о JSON

JSON сегодня очень распространён и представляет собой формат данных, представленных в текстовом виде. За счёт своей универсальности и простоты JSON прекрасно работает на различных платформах. С его помощью разработчик может выполнять сериализацию структур информации в целях последующей передачи данных между приложениями. Как пример — обмен текстовой информацией между сервером и браузером о клиентах в интернет-магазине.

Данные в формате JSON могут быть представлены в нескольких видах: 1) последовательность пар с ключами и соответствующими этим ключам значениями; 2) упорядоченный набор значений.

Значения, передаваемые в JSON, могут быть строками, числами, объектами, литералами (true, false, null), одномерными массивами. Что касается Python, то он поддерживает работу с JSON-форматом с помощью специального json-модуля и методов по кодированию/декодированию данных. В результате можно получать и отправлять сведения в виде, комфортном для чтения.

Сохраняем данные в JSON в Python

Если мы хотим записать информацию в JSON-формате, используя средства языка программирования Python, для начала надо подключить соответствующий json-модуль. Для этого нам пригодиться команда import json в самом начале кода.

Также стоит упомянуть метод dumps — он отвечает за автоматическую упаковку информации в JSON и принимает переменную, содержащую все необходимые данные.

Теперь давайте продемонстрируем кодирование словаря dictData. В нём содержатся некоторые данные о пользователе интернет-портала: идентификационный код, пароль, логин, имя, номер телефона, информация об активности, e-mail. Все эти значения представлены в форме обыкновенных строк, а также булевых литералов True/False и целых чисел. Вот наш пример:

Выполнив метод dumps , мы получим результат, который передастся в переменную с названием jsonData. То есть мы видим, что словарь dictData преобразовался в формат JSON всего лишь одной строчкой. А за счёт функции print вся информация была закодирована в изначальном виде. Также следует добавить, что сведения из поля online преобразовались из литерала True в true.

Теперь, используя Python, выполним запись json в файл. Чтобы это сделать, дополним предыдущий код:

Разбираем JSON-данные в Python

Если мы хотим выполнить обратную операцию и быстро раскодировать формат JSON средствами языка Python, нам поможет метод loads . Он позволяет без труда преобразовать JSON в объект, и с этим объектом мы сможем легко взаимодействовать в программе.

В нашем следующем примере мы продемонстрируем создание аналогичного JSON-объекта с имеющейся информацией о пользователе. Если мы будем в качестве параметра передавать переменную jsonData методу loads , на выходе получим словарь dictData, а из него уже сможем получить нужные данные. Print выведет отдельные поля dictData: имя, информацию об активности, номер телефона, адрес e-mail.

Мы видим, что произошло обратное, а литерал true автоматически преобразовался в True. Это произошло, чтобы была возможность работать с ним средствами Python.

Работа с JSON в Python

В этом руководстве мы будем обсуждать JSON в Python: как кодировать и декодировать данные. Прежде чем начать работу с json-модулем Python, мы сначала обсудим данные JSON.

Аббревиатура JSON – это нотация объектов JavaScript. Согласно Википедии, JSON – это формат файла открытого стандарта, который использует читаемый человеком текст для передачи объектов данных, состоящих из пар атрибут-значение и типов данных массива (или любого другого сериализуемого значения).

JSON – очень распространенный формат данных, используемый для асинхронной связи между браузером и сервером. Правила синтаксиса для JSON приведены ниже:

1. Данные – это просто пара имени и значения.
2. Данные, объект и массивы разделяются запятыми.
3. Фигурные скобки удерживают объект.
4. Квадрат содержит массив.

json.dumps()

В этом разделе мы узнаем, как преобразовать данные Python в данные JSON. Задача очень простая. Сначала импортируйте модуль json. Затем используйте функцию json.dumps() для декодирования данных json. Ниже приведен простой пример функции json.dumps() в Python.

Вы получите такой результат.

Pretty print

Как вы можете видеть в приведенном выше примере, для json pretty print мы должны передать дополнительную переменную indent функции json dumps. Например, json.dumps (nested_list, indent = 2).

json.loads()

Вы можете легко преобразовать данные JSON в объекты Python. Используя функцию json.loads(), вы можете просто преобразовать данные JSON в данные Python. Итак, посмотрите следующий пример кода parse json, чтобы понять функцию загрузки.

Ниже приведен результат работы примера программы синтаксического анализа json на Python.

Преобразование данных JSON

В предыдущих двух разделах вы могли заметить, что список Python преобразуется в данные JSONArray, а словарь Python становится JSONObject. Итак, какой объект Python по умолчанию преобразован в объект JSON, показан в таблице ниже.

dict	object
list, tuple	array
str	string
int, float, int — производные от float перечисления	number
True	true
False	false
None	null

Кроме того, если вы конвертируете JSONArray, вы получите список. Так что здесь тоже есть некоторые правила. Итак, в следующих таблицах показан тип данных JSON, которые преобразуются в данные.

Working with JSON data in Python

In this tutorial, we will discuss Working with JSON data in Python. Also, We will see these below topics as:

• JSON with python
• Python extract data from JSON file
• Extract specific data from JSON python
• Read JSON file python
• Python read JSON file line by line
• Python object to JSON
• Python create JSON array
• Python write JSON to file pretty
• Python string to JSON
• Project using JSON

Table of Contents

json with python

• JSON stands for JavaScript Object Notation.
• JSON is popular because of its lightweight data-interchange format.
• JSON format looks like a dictionary in python
• It is in a key and value format
• JSON module always produce str objects.
• dumps() : converting Python to JSON
• loads() : converting JSON to Python

Python extract data from JSON file

In this section, we will learn how to extract data from JSON file in python.

Step 1: import json module

Step 2: Download dateset

Step 3: read file using open() and store it in f variable

Step 4: Parse f into python object using load().

Step 5: pass ‘Key‘ ‘index-number’ ‘key’& print the information.

ode: Our objective is to fetch the names of all teachers.

Output:

In this output, we have printed name of all the teachers. db[‘teacher_db’][0][‘name’].
If this is confusing you then check the explanation in the next section (Extract field from JSON python).

Extract specific data from JSON python

Now, let us see how to extract specific data from JSON in Python.

• Extracting information is an art.
• In this section, we will learn how to get what we want.
• While dealing with arrays, pay attention to nested arrays.
• In the previous scenario, we used db[‘teacher_db’][0][‘name’] to get the names of teachers.
• where db is the variable that holds all the values that is why it is placed at the first position.
• Now we have two options. (student_db & teacher_db)
• We choose teacher_db, so we place it in the second position.
• Now we have options (t_id, name, class, isPermanet)
• each has an index value starting from 0 – 3.
• Since we want a name so we provided the index value as [1] (the name is at position 1)
• and then we mentioned key i.e [‘name’] in this case.
• that is why the complete statement becomes db[‘teacher_db’][0][‘name’]
• And this is how we extract the specific value.

Read JSON file python

In this section we will learn how to read json file in python.

Step 1: import the json module

Step 2: Use open() to read the json file and store this information in file variable.

Step 3: convert json to python using load() and store the information in db variable.

Step 4: Print the variable.

Code:

Output:

The output displays all the information in the file, everything is in a key & value pair..

This is how we can read json file data in python.

Python read JSON file line by line

In this section, we will see how to read json file by line in Python and keep on storing it in an empty python list.

Step 1: import json module.

Step 2: Create empty python list with the name lineByLine

Step 3: Read the json file using open() and store the information in file variable.

Step 4: Convert item from json to python using load() & store the information in db variable.

Step 5: append db in lineByLine empty list.

Step 6: start a loop & print item of lineByLine list

Code:

Output:

The output displays the content of json file line by line. Each item is displayed in different lines. The data type of each line is Python dict .

This is how we can read json file line by line in python.

Python object to JSON

Now, let us see how to convert Python object to json.

• JSON is a javascript object. These objects need to be parsed to python objects and then only we can use & manipulate them.
• json.dumps() is used to convert or parse python object to JSON

Code:

Output:

In this output, python object i.e dict has been parsed or converted to json object i.e str.
Also, False & false have been pointed. python has False with uppercase ‘F‘ where has json has lowercase ‘f’.

Python create JSON array

plays a major role in data structuring.
• In this section. we will learn how to create an array & nested array in Python.
• nested array means array inside another array.
• The number of nested-arrays determines the dimension of the object.

Code:

Output:

In this output, multi-dimensional array is created using JSON.

Python write json to file pretty

• Pretty does exactly how it sounds.
• It improves the look of the output. Makes it more readable.
• It displays data with indentation and sorting.
• in JSON by default indent=none & Sort_file=false
• But it can be changed to any value like indent = 2
• And sort_file can be true. This will arrange all the keys in an ascending order.

Code:

Output without prettyprint

In this output you can notice that everything is wrapped in 3 lines, it is not easy to understand.

Output with prettyprint

In this output you can see that it looks good & can be understand easily. All We did to make this data look like this is converted python to json using Dump and provided indentation of 2.

Python string to JSON

• In this section, we will learn how to convert string to JSON in Python
• JSON always return ‘str’
• So, no change will be visible here but it is converted to json.

Code:

Output:

In this output, the string was converted to JSON and JSON always returns str that is why data type is still showing <class ‘str’>

Project using JSON

In this project, we are creating a dictionary using JSON. Users can search for the meaning of any word. In case the word is not available then the program will show an error prompt.

Scope for improvement:

Though the project is complete still there is scope for more features that you can try to add by yourself. In case, you face any problem, write in the comment box.

• section to add new words
• exit button
• improve Gui.

Code:

Output:

In this output, a dictionary application is created. You can search for any meaning of a word. The meaning of the word will be shown using a popup message box. In case the word is not available error pop-up message will appear.

In case, you the word is not available in the JSON file then you will will see error message.

You may like the following Python tutorials:

We have learned these:

• JSON with python
• Python extract data from JSON file
• Extract specific data from JSON python
• Read JSON file python
• Python read JSON file line by line
• Python object to JSON
• Python create JSON array
• Python write JSON to file pretty
• Python string to JSON
• Project using JSON

I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.